2024-02-13T14:53:53.292 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-13T14:53:53.292 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098 --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --owner scheduled_phlogistonjohn@teuthology --name phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi -- /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/orig.config.yaml 2024-02-13T14:53:53.324 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T14:53:53.451 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098 branch: wip-phlogistonjohn-testing-2024-02-12-1654 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: jmulligan@redhat.com first_in_suite: false job_id: '7558098' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-02-12-1654 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 workunit: branch: wip-phlogistonjohn-testing-2024-02-12-1654 sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 owner: scheduled_phlogistonjohn@teuthology priority: 99 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1588 sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 sleep_before_teardown: 0 subset: 1/15 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-02-12-1654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPAoKKIJhqZ3r5YochdFAD9JRIWbhb0onyuXid4rRVyBCSwGUDpdSuMAroxNUke3UW6SMTgpdJlt0QFrArmlvc= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIUjl3gaY9DGwFloYSKxb6NtjiX9PjiiG4Bz2Ehh0H1cc2C2oMtZZsKe6wXj4ec03EF9kN+bGZbKmfXlgVXXmrY= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-13_11:41:43 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-13T14:53:53.452 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa; will attempt to use it 2024-02-13T14:53:53.452 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa/tasks 2024-02-13T14:53:53.453 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-13T14:53:53.454 INFO:teuthology.task.internal:Checking packages... 2024-02-13T14:53:53.478 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e1b79d5a4193a484176863ab96c0ca28c00e1662' 2024-02-13T14:53:53.478 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-13T14:53:53.478 INFO:teuthology.packaging:ref: None 2024-02-13T14:53:53.479 INFO:teuthology.packaging:tag: None 2024-02-13T14:53:53.479 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-02-12-1654 2024-02-13T14:53:53.479 INFO:teuthology.packaging:sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:53:53.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-02-12-1654 2024-02-13T14:53:53.618 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1423.ge1b79d5a 2024-02-13T14:53:53.620 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-13T14:53:53.627 INFO:teuthology.task.internal:no buildpackages task found 2024-02-13T14:53:53.627 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-13T14:53:53.647 INFO:teuthology.task.internal:Saving configuration 2024-02-13T14:53:53.659 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-13T14:53:53.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T14:53:53.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:48:57.730978', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPAoKKIJhqZ3r5YochdFAD9JRIWbhb0onyuXid4rRVyBCSwGUDpdSuMAroxNUke3UW6SMTgpdJlt0QFrArmlvc='} 2024-02-13T14:53:53.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:48:57.729610', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIUjl3gaY9DGwFloYSKxb6NtjiX9PjiiG4Bz2Ehh0H1cc2C2oMtZZsKe6wXj4ec03EF9kN+bGZbKmfXlgVXXmrY='} 2024-02-13T14:53:53.715 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-13T14:53:53.722 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-13T14:53:53.722 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-13T14:53:53.722 INFO:teuthology.run_tasks:Running task console_log... 2024-02-13T14:53:53.800 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feaedd560d0>, signals=[15]) 2024-02-13T14:53:53.801 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-13T14:53:53.807 INFO:teuthology.task.internal:Opening connections... 2024-02-13T14:53:53.807 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T14:53:53.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:53:53.915 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-02-13T14:53:53.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:53:54.003 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-13T14:53:54.010 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-02-13T14:53:54.027 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-02-13T14:53:54.028 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-02-13T14:53:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-13T14:53:54.085 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T14:53:54.086 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-02-13T14:53:54.112 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-02-13T14:53:54.129 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-02-13T14:53:54.130 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-02-13T14:53:54.185 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-02-13T14:53:54.185 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-02-13T14:53:54.185 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-13T14:53:54.186 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T14:53:54.186 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-02-13T14:53:54.211 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-13T14:53:54.220 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-13T14:53:54.229 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-13T14:53:54.229 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-02-13T14:53:54.232 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-02-13T14:53:54.247 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-13T14:53:54.253 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-13T14:53:54.253 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-02-13T14:53:54.288 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-02-13T14:53:54.315 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-13T14:53:54.366 INFO:teuthology.run_tasks:Running task kernel... 2024-02-13T14:53:54.379 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-13T14:53:54.379 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-13T14:53:54.380 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T14:53:54.380 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T14:53:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:53:54.396 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-02-13T14:53:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:53:54.397 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-02-13T14:53:54.452 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-02-13T14:53:54.452 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-02-13T14:53:54.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-02-13T14:53:54.456 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-361.el9.x86_64 2024-02-13T14:53:54.456 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.14.0-361.el9.x86_64 2024-02-13T14:53:54.456 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2024-02-13T14:53:57.101 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-02-13T14:53:57.245 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 6.3 MB/s | 6.7 MB 00:01 2024-02-13T14:53:58.776 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-02-13T14:53:58.825 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-02-13T14:53:59.492 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.0 MB 00:00 2024-02-13T14:54:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-02-13T14:54:03.157 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-02-13T14:54:03.986 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-02-13T14:54:08.093 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.9 MB 00:01 2024-02-13T14:54:09.719 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-02-13T14:54:09.751 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 5.9 MB 00:02 2024-02-13T14:54:11.484 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-02-13T14:54:13.003 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-13T14:54:16.712 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 20 MB 00:06 2024-02-13T14:54:18.069 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-02-13T14:54:20.434 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-13T14:54:20.434 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-13T14:54:20.525 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-13T14:54:20.532 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:54:20.533 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 99 M 2024-02-13T14:54:20.534 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 122 M 2024-02-13T14:54:20.534 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-13T14:54:21.553 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-02-13T14:54:21.737 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 5.0 MB/s | 6.0 MB 00:01 2024-02-13T14:54:23.860 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-13T14:54:23.860 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-13T14:54:23.965 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-02-13T14:54:23.972 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-02-13T14:54:23.973 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-02-13T14:54:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:54:23.973 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 99 M 2024-02-13T14:54:23.973 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 122 M 2024-02-13T14:54:23.973 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-02-13T14:54:24.445 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 5.3 MB/s | 21 MB 00:03 2024-02-13T14:54:25.396 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 4.2 MB/s | 6.0 MB 00:01 2024-02-13T14:54:25.637 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 8.6 MB/s | 33 MB 00:03 2024-02-13T14:54:26.213 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 9.3 MB/s | 21 MB 00:02 2024-02-13T14:54:27.338 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-02-13T14:54:27.647 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 15 MB/s | 33 MB 00:02 2024-02-13T14:54:27.650 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:54:27.651 INFO:teuthology.orchestra.run.smithi098.stdout:Total 27 MB/s | 99 MB 00:03 2024-02-13T14:54:27.651 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-02-13T14:54:28.002 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-02-13T14:54:28.002 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-02-13T14:54:29.030 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-02-13T14:54:29.030 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-02-13T14:54:29.220 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 4.5 MB/s | 39 MB 00:08 2024-02-13T14:54:29.223 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:54:29.223 INFO:teuthology.orchestra.run.smithi002.stdout:Total 11 MB/s | 99 MB 00:08 2024-02-13T14:54:29.224 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-13T14:54:29.671 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-13T14:54:29.671 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-13T14:54:30.742 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-13T14:54:30.743 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-13T14:54:31.428 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-02-13T14:54:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:54:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:54:33.264 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-13T14:54:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:54:34.062 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:54:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:54:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:54:36.951 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:54:37.138 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:54:39.866 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:54:40.115 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:54:42.208 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:54:42.303 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:54:44.819 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:54:47.390 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:14.069 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:14.069 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-13T14:55:14.069 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:55:14.138 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:55:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:55:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-02-13T14:55:15.431 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:55:15.559 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:15.560 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:55:15.560 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-13T14:55:15.560 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T14:55:15.560 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T14:55:15.560 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-13T14:55:15.560 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T14:55:15.561 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:55:15.667 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:15.667 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:55:15.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-02-13T14:55:15.712 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-02-13T14:55:15.712 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-02-13T14:55:15.713 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T14:55:15.714 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T14:55:15.715 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T14:55:15.715 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-419.el9.x86_64 2024-02-13T14:55:15.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T14:55:16.278 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-02-13T14:55:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-13T14:55:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-02-13T14:55:17.693 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-02-13T14:55:17.708 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.qHE8wx1Zl1 2024-02-13T14:55:17.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qHE8wx1Zl1 2024-02-13T14:55:17.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.qHE8wx1Zl1 2024-02-13T14:55:17.909 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.qHE8wx1Zl1 is 6KB 2024-02-13T14:55:17.940 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:17.941 INFO:teuthology.orchestra.run.smithi002.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-13T14:55:17.941 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:55:17.959 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.qHE8wx1Zl1 2024-02-13T14:55:17.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T14:55:18.014 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:18.038 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-13T14:55:18.038 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-13T14:55:18.038 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-13T14:55:18.038 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-13T14:55:18.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-13T14:55:18.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-02-13T14:55:18.460 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:18.460 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:55:18.460 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:55:18.461 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:55:18.891 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:55:18.892 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-13T14:55:19.382 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:55:19.521 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:19.521 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:55:19.521 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-13T14:55:19.521 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T14:55:19.521 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T14:55:19.522 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-13T14:55:19.522 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T14:55:19.522 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:55:19.635 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:55:19.635 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:55:19.635 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 68.el9 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8264254 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-13T14:55:19.676 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T14:55:19.677 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T14:55:19.678 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T14:55:19.678 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.14.0-419.el9.x86_64 2024-02-13T14:55:19.678 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T14:55:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-02-13T14:55:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-13T14:55:21.754 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-02-13T14:55:21.756 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-02-13T14:55:21.771 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.D3YzGFnBH0 2024-02-13T14:55:21.772 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D3YzGFnBH0 2024-02-13T14:55:21.844 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.D3YzGFnBH0 2024-02-13T14:55:21.990 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.D3YzGFnBH0 is 6KB 2024-02-13T14:55:22.010 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.D3YzGFnBH0 2024-02-13T14:55:22.028 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T14:55:22.096 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-13T14:55:22.096 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-13T14:55:22.096 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-13T14:55:22.096 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-13T14:55:22.097 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-13T14:55:22.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-02-13T14:55:48.204 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-13T14:55:48.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T14:55:48.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:55:52.268 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-13T14:55:52.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-13T14:55:52.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:56:22.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-02-13T14:56:31.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T14:56:31.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:56:50.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-02-13T14:56:52.272 DEBUG:teuthology.orchestra.remote:timed out 2024-02-13T14:57:01.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-13T14:57:01.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:57:01.638 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-02-13T14:57:02.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-13T14:57:02.048 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-13T14:57:02.048 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-02-13T14:57:02.062 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-419.el9.x86_64 2024-02-13T14:57:02.062 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-13T14:57:02.063 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T14:57:02.063 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T14:57:02.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T14:57:02.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:57:02.609 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-02-13T14:57:03.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T14:57:03.021 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-13T14:57:03.021 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-02-13T14:57:03.036 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-419.el9.x86_64 2024-02-13T14:57:03.036 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-13T14:57:03.036 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T14:57:03.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T14:57:03.064 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-13T14:57:03.064 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T14:57:03.116 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-02-13T14:57:03.151 DEBUG:teuthology.parallel:result is None 2024-02-13T14:57:04.037 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-13T14:57:04.037 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T14:57:04.127 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-02-13T14:57:04.171 DEBUG:teuthology.parallel:result is None 2024-02-13T14:57:04.172 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-13T14:57:04.182 INFO:teuthology.task.internal:Creating test directory... 2024-02-13T14:57:04.182 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T14:57:04.184 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T14:57:04.199 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-13T14:57:04.204 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-13T14:57:04.213 INFO:teuthology.task.internal:Creating archive directory... 2024-02-13T14:57:04.214 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T14:57:04.243 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T14:57:04.272 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-13T14:57:04.279 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-13T14:57:04.279 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T14:57:04.303 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T14:57:04.338 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:57:04.351 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:57:04.357 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:57:04.369 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:57:04.371 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-13T14:57:04.377 INFO:teuthology.task.internal:Configuring sudo... 2024-02-13T14:57:04.377 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T14:57:04.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T14:57:04.422 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-13T14:57:04.431 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-13T14:57:04.431 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T14:57:04.450 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T14:57:04.480 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:57:04.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:57:04.601 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:57:04.683 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:57:04.730 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T14:57:04.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T14:57:04.800 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:57:04.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:57:04.924 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:57:04.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:57:05.022 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T14:57:05.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T14:57:05.108 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-02-13T14:57:05.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-02-13T14:57:05.172 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:57:05.354 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:57:05.421 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-13T14:57:05.429 INFO:teuthology.task.internal:Starting timer... 2024-02-13T14:57:05.429 INFO:teuthology.run_tasks:Running task pcp... 2024-02-13T14:57:05.438 INFO:teuthology.run_tasks:Running task selinux... 2024-02-13T14:57:05.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-02-13T14:57:05.547 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-02-13T14:57:05.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-02-13T14:57:05.628 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-02-13T14:57:05.633 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-13T14:57:05.633 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-02-13T14:57:05.785 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-02-13T14:57:05.786 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-02-13T14:57:05.818 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-02-13T14:57:05.818 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-02-13T14:57:05.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:57:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:57:05.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:57:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:57:05.918 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-13T14:57:05.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-02-13T14:57:05.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-02-13T14:57:06.012 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-13T14:57:06.022 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-13T14:57:06.037 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-13T14:57:06.038 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-13T15:01:23.077 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-02-13T15:01:23.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-13T15:01:23.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T15:01:23.170 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-02-13T15:01:23.250 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-13T15:01:23.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T15:01:23.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T15:01:23.336 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-02-13T15:01:23.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-02-13T15:01:23.425 INFO:teuthology.run_tasks:Running task clock... 2024-02-13T15:01:23.436 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-13T15:01:23.436 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T15:01:23.437 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T15:01:23.439 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T15:01:23.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T15:01:23.472 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T15:01:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T15:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T15:01:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T15:01:23.520 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-02-13T15:01:23.535 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-02-13T15:01:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-02-13T15:01:23.552 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T15:01:23.559 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-02-13T15:01:23.568 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T15:01:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T15:01:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T15:01:23.620 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.624 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.643 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-02-13T15:01:23.646 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T15:01:23.646 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-02-13T15:01:23.646 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.646 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.647 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.647 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:01:23.647 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-13T15:01:23.746 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e1b79d5a4193a484176863ab96c0ca28c00e1662'} 2024-02-13T15:01:23.746 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-13T15:01:23.747 INFO:tasks.cephadm:Cluster fsid is c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:01:23.747 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-13T15:01:23.747 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.c': '[v2:172.21.15.2:3301,v1:172.21.15.2:6790]', 'mon.b': '172.21.15.98'} 2024-02-13T15:01:23.747 INFO:tasks.cephadm:First mon is mon.a on smithi002 2024-02-13T15:01:23.747 INFO:tasks.cephadm:First mgr is y 2024-02-13T15:01:23.747 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-13T15:01:23.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2024-02-13T15:01:23.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-02-13T15:01:23.803 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-13T15:01:23.803 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.081 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 13 15:01 /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.081 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.257 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 13 15:01 /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.257 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.273 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T15:01:24.298 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-13T15:01:24.298 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-13T15:01:24.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-13T15:01:25.231 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-13T15:01:25.252 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-13T15:01:47.706 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-02-13T15:01:47.706 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-02-13T15:01:47.707 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-13T15:01:49.818 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-02-13T15:01:49.819 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:01:49.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2024-02-13T15:01:49.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-02-13T15:01:49.919 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2024-02-13T15:01:49.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-02-13T15:01:49.989 INFO:tasks.cephadm:Writing seed config... 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-13T15:01:49.990 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-13T15:01:49.991 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-13T15:01:49.991 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-13T15:01:49.991 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:01:49.991 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-13T15:01:50.005 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c0fca16a-ca80-11ee-95b9-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-13T15:01:50.006 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service 2024-02-13T15:01:50.048 DEBUG:teuthology.orchestra.run.smithi002:mgr.y> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service 2024-02-13T15:01:50.090 INFO:tasks.cephadm:Bootstrapping... 2024-02-13T15:01:50.090 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:01:50.290 INFO:teuthology.orchestra.run.smithi002.stderr:-------------------------------------------------------------------------------- 2024-02-13T15:01:50.290 INFO:teuthology.orchestra.run.smithi002.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'c0fca16a-ca80-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2024-02-13T15:01:50.312 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 4.9.3-dev 2024-02-13T15:01:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying podman|docker is present... 2024-02-13T15:01:50.335 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 4.9.3-dev 2024-02-13T15:01:50.338 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying lvm2 is present... 2024-02-13T15:01:50.338 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying time synchronization is in place... 2024-02-13T15:01:50.344 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T15:01:50.350 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:01:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2024-02-13T15:01:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2024-02-13T15:01:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2024-02-13T15:01:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:Repeating the final host check... 2024-02-13T15:01:50.384 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 4.9.3-dev 2024-02-13T15:01:50.386 INFO:teuthology.orchestra.run.smithi002.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-13T15:01:50.386 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl is present 2024-02-13T15:01:50.386 INFO:teuthology.orchestra.run.smithi002.stderr:lvcreate is present 2024-02-13T15:01:50.392 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T15:01:50.398 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:01:50.404 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2024-02-13T15:01:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2024-02-13T15:01:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2024-02-13T15:01:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:Host looks OK 2024-02-13T15:01:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:Cluster fsid: c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:01:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:Acquiring lock 140548761783792 on /run/cephadm/c0fca16a-ca80-11ee-95b9-87774f69a715.lock 2024-02-13T15:01:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:Lock 140548761783792 acquired on /run/cephadm/c0fca16a-ca80-11ee-95b9-87774f69a715.lock 2024-02-13T15:01:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 3300 ... 2024-02-13T15:01:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 6789 ... 2024-02-13T15:01:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:Base mon IP is 172.21.15.2, final addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-02-13T15:01:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.2 metric 100 2024-02-13T15:01:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2024-02-13T15:01:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-13T15:01:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-13T15:01:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-02-13T15:01:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T15:01:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-02-13T15:01:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-13T15:01:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-13T15:01:50.525 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-13T15:01:51.665 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Getting image source signatures 2024-02-13T15:01:51.665 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-13T15:01:51.665 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-13T15:01:51.665 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-13T15:01:51.666 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-13T15:01:51.666 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-13T15:01:51.666 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-13T15:01:51.812 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Writing manifest to image destination 2024-02-13T15:01:51.974 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-13T15:01:52.357 INFO:teuthology.orchestra.run.smithi002.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-13T15:01:52.872 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-13T15:01:52.872 INFO:teuthology.orchestra.run.smithi002.stderr:Extracting ceph user uid/gid from container image... 2024-02-13T15:01:53.239 INFO:teuthology.orchestra.run.smithi002.stderr:stat: 167 167 2024-02-13T15:01:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial keys... 2024-02-13T15:01:54.119 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQBihMtlWzkoAhAAm+qR2rAiF/hTQtWB3jsJ4A== 2024-02-13T15:01:55.016 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQBihMtlqImXNxAATN0pTXhRf5H56vIePjkwaA== 2024-02-13T15:01:55.921 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQBjhMtle3BeNBAAz0l0q+Fh4SB7B/N7L5MP4A== 2024-02-13T15:01:56.495 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial monmap... 2024-02-13T15:01:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T15:01:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-13T15:01:56.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:01:56.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T15:01:57.374 INFO:teuthology.orchestra.run.smithi002.stderr:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T15:01:57.374 INFO:teuthology.orchestra.run.smithi002.stderr:setting min_mon_release = octopus 2024-02-13T15:01:57.374 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: set fsid to c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:01:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T15:01:57.375 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:01:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mon... 2024-02-13T15:01:58.281 INFO:teuthology.orchestra.run.smithi002.stderr:create mon.a on 2024-02-13T15:01:58.490 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-13T15:01:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target. 2024-02-13T15:01:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target. 2024-02-13T15:01:58.935 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-13T15:01:58.942 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target.wants/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@.service. 2024-02-13T15:01:59.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:01:59 smithi002 podman[24391]: 2024-02-13 15:01:59.549044229 +0000 UTC m=+0.336609407 container start 8b135aeb28d9b666e64517ccb751b00458f99fbe370d59b32818d04de0bf0381 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.buildah.version=1.19.8, version=8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vendor=Red Hat, Inc., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.expose-services=) 2024-02-13T15:01:59.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:01:59 smithi002 ceph-mon[24409]: mkfs c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:01:59.660 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-02-13T15:01:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:01:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-02-13T15:01:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T15:01:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon to start... 2024-02-13T15:01:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon... 2024-02-13T15:01:59.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:01:59 smithi002 ceph-mon[24409]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T15:01:59.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:01:59 smithi002 bash[24391]: 8b135aeb28d9b666e64517ccb751b00458f99fbe370d59b32818d04de0bf0381 2024-02-13T15:01:59.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:01:59 smithi002 systemd[1]: Started Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: cluster: 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: id: c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: services: 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.500646s) 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: data: 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-13T15:02:00.111 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-13T15:02:00.112 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pgs: 2024-02-13T15:02:00.112 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:00.565 INFO:teuthology.orchestra.run.smithi002.stderr:mon is available 2024-02-13T15:02:00.565 INFO:teuthology.orchestra.run.smithi002.stderr:Assimilating anything we can from ceph.conf... 2024-02-13T15:02:00.873 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T15:02:00.873 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2024-02-13T15:02:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: fsmap 2024-02-13T15:02:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: osdmap e1: 0 total, 0 up, 0 in 2024-02-13T15:02:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: mgrmap e1: no daemons active 2024-02-13T15:02:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:00 smithi002 ceph-mon[24409]: from='client.? 172.21.15.2:0/3102035159' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-13T15:02:01.050 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-13T15:02:01.051 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-13T15:02:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:Generating new minimal ceph.conf... 2024-02-13T15:02:01.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:01 smithi002 ceph-mon[24409]: from='client.? 172.21.15.2:0/3368130339' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T15:02:01.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:01 smithi002 ceph-mon[24409]: from='client.? 172.21.15.2:0/3368130339' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T15:02:02.596 INFO:teuthology.orchestra.run.smithi002.stderr:Restarting the monitor... 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 systemd[1]: Stopping Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 ceph-mon[24409]: from='client.? 172.21.15.2:0/3576845861' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 bash[24626]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[24405]: 2024-02-13T15:02:02.734+0000 7f6babd2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[24405]: 2024-02-13T15:02:02.734+0000 7f6babd2d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-13T15:02:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:02 smithi002 podman[24637]: 2024-02-13 15:02:02.744329312 +0000 UTC m=+0.096115654 container died 8b135aeb28d9b666e64517ccb751b00458f99fbe370d59b32818d04de0bf0381 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, GIT_CLEAN=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vcs-type=git, name=centos-stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24637]: 2024-02-13 15:02:03.150289578 +0000 UTC m=+0.502075916 container cleanup 8b135aeb28d9b666e64517ccb751b00458f99fbe370d59b32818d04de0bf0381 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_CLEAN=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 bash[24637]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 bash[24663]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24651]: 2024-02-13 15:02:03.317725467 +0000 UTC m=+0.573704796 container remove 8b135aeb28d9b666e64517ccb751b00458f99fbe370d59b32818d04de0bf0381 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, release=754, name=centos-stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 systemd[1]: Stopped Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 systemd[1]: Starting Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:02:03.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24735]: 2024-02-13 15:02:03.535117957 +0000 UTC m=+0.099639799 container create 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-13T15:02:03.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24735]: 2024-02-13 15:02:03.454179168 +0000 UTC m=+0.018700991 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-13T15:02:03.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24735]: 2024-02-13 15:02:03.720691923 +0000 UTC m=+0.285213733 container init 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-type=git) 2024-02-13T15:02:03.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 podman[24735]: 2024-02-13 15:02:03.723462877 +0000 UTC m=+0.287984673 container start 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , version=8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, name=centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=) 2024-02-13T15:02:03.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: pidfile_write: ignore empty --pid-file 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: load: jerasure load: lrc 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: RocksDB version: 6.15.5 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Compile date Apr 18 2022 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: DB SUMMARY 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: DB Session ID: DW5OPSQQ573D6A70NK4S 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: CURRENT file: CURRENT 2024-02-13T15:02:03.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82607 ; 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.error_if_exists: 0 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.create_if_missing: 0 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.paranoid_checks: 1 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.env: 0x55f3240c6860 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.fs: Posix File System 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.info_log: 0x55f325cbfdc0 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.statistics: (nil) 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.use_fsync: 0 2024-02-13T15:02:03.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_log_file_size: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_fallocate: 1 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.use_direct_reads: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.db_log_dir: 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T15:02:03.808 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T15:02:03.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.write_buffer_manager: 0x55f325db02a0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.unordered_write: 0 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T15:02:03.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.row_cache: None 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.wal_filter: None 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.preserve_deletes: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.two_write_queues: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.atomic_flush: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.log_readahead_size: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T15:02:03.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_background_jobs: 2 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_background_compactions: -1 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_subcompactions: 1 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_open_files: -1 2024-02-13T15:02:03.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_background_flushes: -1 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Compression algorithms supported: 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kZSTD supported: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kXpressCompression supported: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kLZ4Compression supported: 1 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kBZip2Compression supported: 0 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kZlibCompression supported: 1 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: kSnappyCompression supported: 1 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-02-13T15:02:03.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.merge_operator: 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_filter: None 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f325c8bd00) 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-02-13T15:02:03.814 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: hash_index_allow_collision: 1 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 1 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55f325cf6170 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-02-13T15:02:03.815 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 4 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T15:02:03.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression: NoCompression 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.num_levels: 7 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T15:02:03.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T15:02:03.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T15:02:03.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T15:02:03.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T15:02:03.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:02:03.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T15:02:03.818 INFO:teuthology.orchestra.run.smithi002.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T15:02:03.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.arena_block_size: 4194304 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T15:02:03.820 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.table_properties_collectors: 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.inplace_update_support: 0 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.bloom_locality: 0 2024-02-13T15:02:03.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.max_successive_merges: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.ttl: 2592000 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.enable_blob_files: false 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.min_blob_size: 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-13T15:02:03.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707836523750270, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707836523750976, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79157, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77420, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71729, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707836523, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e10bc867-dd77-4abd-8e5f-be9af9b1ccdd", "db_session_id": "DW5OPSQQ573D6A70NK4S"}} 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707836523751298, "job": 1, "event": "recovery_finished"} 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55f325cdca80 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: DB pointer 0x55f325cec000 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:02:03.823 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:02:03.824 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 25.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 25.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T15:02:03.825 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.9 0.00 0.00 1 0.000 0 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T15:02:03.826 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 25.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???) e1 preinit fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).mds e1 new map 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).mds e1 print_map 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:02:03.827 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 bash[24735]: 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d 2024-02-13T15:02:03.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 systemd[1]: Started Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:02:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T15:02:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2024-02-13T15:02:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: fsmap 2024-02-13T15:02:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: osdmap e1: 0 total, 0 up, 0 in 2024-02-13T15:02:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:03 smithi002 ceph-mon[24754]: mgrmap e1: no daemons active 2024-02-13T15:02:04.872 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-13T15:02:04.873 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:02:04.873 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mgr... 2024-02-13T15:02:04.873 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying port 9283 ... 2024-02-13T15:02:05.047 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-13T15:02:05.054 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715.target.wants/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@.service. 2024-02-13T15:02:05.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:05 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2150475989' entity='client.admin' 2024-02-13T15:02:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-02-13T15:02:05.729 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:02:05.730 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-02-13T15:02:05.730 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T15:02:05.739 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-02-13T15:02:05.748 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:02:05.748 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-02-13T15:02:05.748 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-13T15:02:05.748 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr to start... 2024-02-13T15:02:05.749 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr... 2024-02-13T15:02:05.865 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:05 smithi002 systemd[1]: Started Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:02:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "c0fca16a-ca80-11ee-95b9-87774f69a715", 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-13T15:02:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T15:02:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T15:02:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2024-02-13T15:02:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-02-13T15:01:59.607679+0000", 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T15:02:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:06 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1830097970' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T15:02:06.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:06.122+0000 7fc2c2afd000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:02:06.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:06.216+0000 7fc2c2afd000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:02:07.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:06.835+0000 7fc2c2afd000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:02:07.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:06.926+0000 7fc2c2afd000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:02:07.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.010+0000 7fc2c2afd000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:02:07.447 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.160+0000 7fc2c2afd000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:02:07.447 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.242+0000 7fc2c2afd000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:02:07.447 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.357+0000 7fc2c2afd000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:07.447 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.445+0000 7fc2c2afd000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:02:07.705 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.613+0000 7fc2c2afd000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:02:08.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:07.703+0000 7fc2c2afd000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:02:09.110 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:08.783+0000 7fc2c2afd000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:02:09.110 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:08.915+0000 7fc2c2afd000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:02:09.110 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.108+0000 7fc2c2afd000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:02:09.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.197+0000 7fc2c2afd000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:02:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.335+0000 7fc2c2afd000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:09.971 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.658+0000 7fc2c2afd000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:02:09.972 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.883+0000 7fc2c2afd000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:09.972 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:09.969+0000 7fc2c2afd000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:10.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:10.143+0000 7fc2c2afd000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:02:10.422 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (1/15)... 2024-02-13T15:02:11.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:11.084+0000 7fc2c2afd000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:02:11.638 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:11.385+0000 7fc2c2afd000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:02:11.639 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:11.482+0000 7fc2c2afd000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:02:11.639 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:11.636+0000 7fc2c2afd000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: Activating manager daemon y 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: mgrmap e2: y(active, starting, since 0.00266419s) 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: Manager daemon y is now available 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:02:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:02:12.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' 2024-02-13T15:02:12.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' 2024-02-13T15:02:12.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:11 smithi002 ceph-mon[24754]: from='mgr.14100 172.21.15.2:0/1973888158' entity='mgr.y' 2024-02-13T15:02:13.546 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:13.546 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "c0fca16a-ca80-11ee-95b9-87774f69a715", 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:13.547 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T15:02:13.548 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T15:02:13.549 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-02-13T15:01:59.607679+0000", 2024-02-13T15:02:13.550 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-02-13T15:02:13.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-02-13T15:02:13.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T15:02:13.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:13.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:13 smithi002 ceph-mon[24754]: mgrmap e3: y(active, since 1.00636s) 2024-02-13T15:02:13.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:13 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3462062125' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T15:02:14.166 INFO:teuthology.orchestra.run.smithi002.stderr:mgr is available 2024-02-13T15:02:14.776 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2024-02-13T15:02:14.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-13T15:02:14.778 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-13T15:02:15.297 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling cephadm module... 2024-02-13T15:02:16.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:15 smithi002 ceph-mon[24754]: mgrmap e4: y(active, since 3s) 2024-02-13T15:02:16.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:15 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1412917402' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T15:02:16.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:15 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1412917402' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T15:02:16.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:16 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3598113140' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-13T15:02:16.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setuser ceph since I am not root 2024-02-13T15:02:16.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setgroup ceph since I am not root 2024-02-13T15:02:17.487 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:17.184+0000 7f868b333000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:02:17.487 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:17.280+0000 7f868b333000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:02:17.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3598113140' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-13T15:02:17.821 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-mon[24754]: mgrmap e5: y(active, since 5s) 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T15:02:17.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:18.077 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:17.899+0000 7f868b333000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:02:18.077 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:17 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:17.990+0000 7f868b333000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:02:18.077 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.074+0000 7f868b333000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:02:18.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.226+0000 7f868b333000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:02:18.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.309+0000 7f868b333000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:02:18.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3619615818' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T15:02:18.674 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.420+0000 7f868b333000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:18.674 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.505+0000 7f868b333000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:02:18.674 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.671+0000 7f868b333000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:02:19.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:18.758+0000 7f868b333000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:02:20.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:19.807+0000 7f868b333000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:02:20.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:19.939+0000 7f868b333000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:02:20.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.134+0000 7f868b333000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:02:20.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.222+0000 7f868b333000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:02:20.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.361+0000 7f868b333000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:20.912 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.683+0000 7f868b333000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:02:21.173 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.909+0000 7f868b333000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:21.173 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:20.995+0000 7f868b333000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:21.173 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:21 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:21.170+0000 7f868b333000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:02:21.677 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2024-02-13T15:02:21.677 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 5... 2024-02-13T15:02:22.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:22.129+0000 7f868b333000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:02:22.721 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:22.431+0000 7f868b333000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:02:22.721 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:22.527+0000 7f868b333000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:02:22.721 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:22.681+0000 7f868b333000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:02:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-13T15:02:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T15:02:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: Active manager daemon y restarted 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: Activating manager daemon y 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: osdmap e2: 0 total, 0 up, 0 in 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: mgrmap e6: y(active, starting, since 0.0529336s) 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: Manager daemon y is now available 2024-02-13T15:02:24.065 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:02:24.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:23 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:02:24.373 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:24] ENGINE Bus STARTING 2024-02-13T15:02:24.373 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:24] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:02:24.373 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:24] ENGINE Bus STARTED 2024-02-13T15:02:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 5 is available 2024-02-13T15:02:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:Setting orchestrator backend to cephadm... 2024-02-13T15:02:25.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: mgrmap e7: y(active, since 1.0545s) 2024-02-13T15:02:25.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:24] ENGINE Bus STARTING 2024-02-13T15:02:25.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:24] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:02:25.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:24] ENGINE Bus STARTED 2024-02-13T15:02:25.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:25.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:24 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:26.227 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: value unchanged 2024-02-13T15:02:26.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:25 smithi002 ceph-mon[24754]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:26.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:25 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:26.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:25 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:26.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:25 smithi002 ceph-mon[24754]: mgrmap e8: y(active, since 2s) 2024-02-13T15:02:26.812 INFO:teuthology.orchestra.run.smithi002.stderr:Generating ssh key... 2024-02-13T15:02:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:26 smithi002 ceph-mon[24754]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:27 smithi002 ceph-mon[24754]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:27 smithi002 ceph-mon[24754]: Generating ssh key... 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: Generating public/private rsa key pair. 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: Your identification has been saved in /tmp/tmpth_6m4ki/key. 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: Your public key has been saved in /tmp/tmpth_6m4ki/key.pub. 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: The key fingerprint is: 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: SHA256:CGswFFN10BmB0awJt2ZFhUDPy/RDRNhOTVHthQKkD/Q ceph-c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: The key's randomart image is: 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: +---[RSA 3072]----+ 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | +o.o*X*@*+oo.o | 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | . .. o=O+o o . o| 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | o .o =*oE . ..| 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | o o*+ *. .| 2024-02-13T15:02:28.366 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | oo. S + | 2024-02-13T15:02:28.367 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | . . | 2024-02-13T15:02:28.367 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | | 2024-02-13T15:02:28.367 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | | 2024-02-13T15:02:28.367 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: | | 2024-02-13T15:02:28.367 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: +----[SHA256]-----+ 2024-02-13T15:02:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:29 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:29 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:29.235 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFmoKYl7na/Tzv9E8G5cvF1pErbIkSQ4J/DRbHl6kd1LQG1+MFPAdzL6dvjdQqSAIrv0Y8OEsZvAy+z/4PCa9ZpYCYQn0vwXs3aq2oqRluTAKpyr7qswtmzpwRZ7MT5IAFuJfBaLVfuzrO0jop9L1iYUmQQt83WuJh/gl3RcFNfBOuujNrFr6aUtOdPe4+paeIukcSUrLaSIteNODtSE5yiliP+5FjvgGHofTwcjQyL4GMOi2m2Uhs/LUzylFgsxA0yLe8oQXFBxDnAddRu7vAVYlfMJmc5ov7ZrFEZVRIPzaFP42RqzZGW0OmkgjJ09gYzGM9rwaIXzoMVOdX67rnrmy9mGuCQq8pdgnPb4ma+5f+X5CllTkDtQYaWIeRgZFfrQYT/zwB2VJmifa73aBpDpYfLO3t3xMV4njpl+XfPAFWL35GMi4B52IuqfAF8aHyUQogo3cm5PuCFUaOxcBJtbCmWqflQsXUWsVTQWztmi4dzfoerz+ME7OtbkFQWsM= ceph-c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:29.817 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-13T15:02:29.818 INFO:teuthology.orchestra.run.smithi002.stderr:Adding key to root@localhost authorized_keys... 2024-02-13T15:02:29.818 INFO:teuthology.orchestra.run.smithi002.stderr:Adding host smithi002... 2024-02-13T15:02:30.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:30 smithi002 ceph-mon[24754]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:31 smithi002 ceph-mon[24754]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:31.417 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Added host 'smithi002' with addr '172.21.15.2' 2024-02-13T15:02:32.115 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mon service... 2024-02-13T15:02:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:32 smithi002 ceph-mon[24754]: Deploying cephadm binary to smithi002 2024-02-13T15:02:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:32 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:32 smithi002 ceph-mon[24754]: Added host smithi002 2024-02-13T15:02:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:32 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:32.743 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-13T15:02:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mgr service... 2024-02-13T15:02:33.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:33 smithi002 ceph-mon[24754]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:33.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:33 smithi002 ceph-mon[24754]: Saving service mon spec with placement count:5 2024-02-13T15:02:33.858 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:33 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:34.068 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-13T15:02:35.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:35 smithi002 ceph-mon[24754]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:35.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:35 smithi002 ceph-mon[24754]: Saving service mgr spec with placement count:2 2024-02-13T15:02:35.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:35 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:36.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:36 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2131732154' entity='client.admin' 2024-02-13T15:02:36.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:36 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:36.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:36 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:37.824 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling the dashboard module... 2024-02-13T15:02:38.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:38 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1260138151' entity='client.admin' 2024-02-13T15:02:39.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1964632134' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-13T15:02:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-mon[24754]: from='mgr.14116 172.21.15.2:0/963301298' entity='mgr.y' 2024-02-13T15:02:39.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setuser ceph since I am not root 2024-02-13T15:02:39.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setgroup ceph since I am not root 2024-02-13T15:02:39.839 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:39.566+0000 7f851aca5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:02:39.839 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:39.658+0000 7f851aca5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:02:40.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1964632134' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-13T15:02:40.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-mon[24754]: mgrmap e9: y(active, since 16s) 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T15:02:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:40.364 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.279+0000 7f851aca5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:02:40.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.373+0000 7f851aca5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:02:40.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.459+0000 7f851aca5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:02:40.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.610+0000 7f851aca5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:02:40.902 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.693+0000 7f851aca5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:02:40.903 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.805+0000 7f851aca5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:40.903 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:40.890+0000 7f851aca5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:02:40.905 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2024-02-13T15:02:40.905 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 9... 2024-02-13T15:02:41.153 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:41 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2234013969' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T15:02:41.154 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:41.063+0000 7f851aca5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:02:41.154 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:41.151+0000 7f851aca5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:02:42.542 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:42.211+0000 7f851aca5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:02:42.542 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:42.344+0000 7f851aca5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:02:42.542 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:42.540+0000 7f851aca5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:02:42.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:42.630+0000 7f851aca5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:02:42.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:42.770+0000 7f851aca5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:43.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:43.105+0000 7f851aca5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:02:43.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:43.331+0000 7f851aca5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:02:43.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:43.419+0000 7f851aca5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:02:43.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:43.594+0000 7f851aca5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:02:44.842 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:44.536+0000 7f851aca5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:02:44.842 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:44.840+0000 7f851aca5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:02:45.093 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:44.937+0000 7f851aca5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:02:45.093 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:45 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:02:45.091+0000 7f851aca5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:02:46.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-02-13T15:02:46.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-13T15:02:46.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T15:02:46.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-02-13T15:02:46.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: Active manager daemon y restarted 2024-02-13T15:02:46.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: Activating manager daemon y 2024-02-13T15:02:46.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: osdmap e3: 0 total, 0 up, 0 in 2024-02-13T15:02:46.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: mgrmap e10: y(active, starting, since 0.052393s) 2024-02-13T15:02:46.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: Manager daemon y is now available 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:02:46.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:02:46.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:46] ENGINE Bus STARTING 2024-02-13T15:02:46.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:46] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:02:46.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:02:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:02:46] ENGINE Bus STARTED 2024-02-13T15:02:46.932 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 9 is available 2024-02-13T15:02:46.932 INFO:teuthology.orchestra.run.smithi002.stderr:Generating a dashboard self-signed certificate... 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: mgrmap e11: y(active, since 1.05583s) 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:46] ENGINE Bus STARTING 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:46] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: [13/Feb/2024:15:02:46] ENGINE Bus STARTED 2024-02-13T15:02:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-13T15:02:48.263 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial admin user... 2024-02-13T15:02:48.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:48.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:48 smithi002 ceph-mon[24754]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:48.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:48 smithi002 ceph-mon[24754]: mgrmap e12: y(active, since 2s) 2024-02-13T15:02:48.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:48.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:49.323 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$TKZgJnfmU0c8rabjV30w0.UYh29znxOAalJSiPcOm.n3ZN1PXkzn6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707836569, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-13T15:02:50.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:49 smithi002 ceph-mon[24754]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:02:50.108 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:50.617 INFO:teuthology.orchestra.run.smithi002.stderr:Fetching dashboard port number... 2024-02-13T15:02:51.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 8443 2024-02-13T15:02:51.877 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-02-13T15:02:51.885 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-02-13T15:02:51.886 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-02-13T15:02:51.886 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-13T15:02:51.888 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph Dashboard is now available at: 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr: URL: https://smithi002.front.sepia.ceph.com:8443/ 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr: User: admin 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr: Password: 721317eok4 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:51.889 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling autotune for osd_memory_target 2024-02-13T15:02:52.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:52 smithi002 ceph-mon[24754]: mgrmap e13: y(active, since 6s) 2024-02-13T15:02:52.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:52 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3338166583' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-13T15:02:54.289 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-13T15:02:54.887 INFO:teuthology.orchestra.run.smithi002.stderr:You can access the Ceph CLI with: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: ceph telemetry on 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr:For more information see: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-13T15:02:54.888 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:02:54.889 INFO:teuthology.orchestra.run.smithi002.stderr:Bootstrap complete. 2024-02-13T15:02:54.928 INFO:tasks.cephadm:Fetching config... 2024-02-13T15:02:54.928 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:02:54.928 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-13T15:02:54.945 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-13T15:02:54.946 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:02:54.946 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-13T15:02:55.002 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-13T15:02:55.002 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:02:55.002 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-13T15:02:55.071 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-13T15:02:55.071 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:02:55.071 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-13T15:02:55.126 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-13T15:02:55.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFmoKYl7na/Tzv9E8G5cvF1pErbIkSQ4J/DRbHl6kd1LQG1+MFPAdzL6dvjdQqSAIrv0Y8OEsZvAy+z/4PCa9ZpYCYQn0vwXs3aq2oqRluTAKpyr7qswtmzpwRZ7MT5IAFuJfBaLVfuzrO0jop9L1iYUmQQt83WuJh/gl3RcFNfBOuujNrFr6aUtOdPe4+paeIukcSUrLaSIteNODtSE5yiliP+5FjvgGHofTwcjQyL4GMOi2m2Uhs/LUzylFgsxA0yLe8oQXFBxDnAddRu7vAVYlfMJmc5ov7ZrFEZVRIPzaFP42RqzZGW0OmkgjJ09gYzGM9rwaIXzoMVOdX67rnrmy9mGuCQq8pdgnPb4ma+5f+X5CllTkDtQYaWIeRgZFfrQYT/zwB2VJmifa73aBpDpYfLO3t3xMV4njpl+XfPAFWL35GMi4B52IuqfAF8aHyUQogo3cm5PuCFUaOxcBJtbCmWqflQsXUWsVTQWztmi4dzfoerz+ME7OtbkFQWsM= ceph-c0fca16a-ca80-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T15:02:55.206 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFmoKYl7na/Tzv9E8G5cvF1pErbIkSQ4J/DRbHl6kd1LQG1+MFPAdzL6dvjdQqSAIrv0Y8OEsZvAy+z/4PCa9ZpYCYQn0vwXs3aq2oqRluTAKpyr7qswtmzpwRZ7MT5IAFuJfBaLVfuzrO0jop9L1iYUmQQt83WuJh/gl3RcFNfBOuujNrFr6aUtOdPe4+paeIukcSUrLaSIteNODtSE5yiliP+5FjvgGHofTwcjQyL4GMOi2m2Uhs/LUzylFgsxA0yLe8oQXFBxDnAddRu7vAVYlfMJmc5ov7ZrFEZVRIPzaFP42RqzZGW0OmkgjJ09gYzGM9rwaIXzoMVOdX67rnrmy9mGuCQq8pdgnPb4ma+5f+X5CllTkDtQYaWIeRgZFfrQYT/zwB2VJmifa73aBpDpYfLO3t3xMV4njpl+XfPAFWL35GMi4B52IuqfAF8aHyUQogo3cm5PuCFUaOxcBJtbCmWqflQsXUWsVTQWztmi4dzfoerz+ME7OtbkFQWsM= ceph-c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:55.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFmoKYl7na/Tzv9E8G5cvF1pErbIkSQ4J/DRbHl6kd1LQG1+MFPAdzL6dvjdQqSAIrv0Y8OEsZvAy+z/4PCa9ZpYCYQn0vwXs3aq2oqRluTAKpyr7qswtmzpwRZ7MT5IAFuJfBaLVfuzrO0jop9L1iYUmQQt83WuJh/gl3RcFNfBOuujNrFr6aUtOdPe4+paeIukcSUrLaSIteNODtSE5yiliP+5FjvgGHofTwcjQyL4GMOi2m2Uhs/LUzylFgsxA0yLe8oQXFBxDnAddRu7vAVYlfMJmc5ov7ZrFEZVRIPzaFP42RqzZGW0OmkgjJ09gYzGM9rwaIXzoMVOdX67rnrmy9mGuCQq8pdgnPb4ma+5f+X5CllTkDtQYaWIeRgZFfrQYT/zwB2VJmifa73aBpDpYfLO3t3xMV4njpl+XfPAFWL35GMi4B52IuqfAF8aHyUQogo3cm5PuCFUaOxcBJtbCmWqflQsXUWsVTQWztmi4dzfoerz+ME7OtbkFQWsM= ceph-c0fca16a-ca80-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T15:02:55.267 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFmoKYl7na/Tzv9E8G5cvF1pErbIkSQ4J/DRbHl6kd1LQG1+MFPAdzL6dvjdQqSAIrv0Y8OEsZvAy+z/4PCa9ZpYCYQn0vwXs3aq2oqRluTAKpyr7qswtmzpwRZ7MT5IAFuJfBaLVfuzrO0jop9L1iYUmQQt83WuJh/gl3RcFNfBOuujNrFr6aUtOdPe4+paeIukcSUrLaSIteNODtSE5yiliP+5FjvgGHofTwcjQyL4GMOi2m2Uhs/LUzylFgsxA0yLe8oQXFBxDnAddRu7vAVYlfMJmc5ov7ZrFEZVRIPzaFP42RqzZGW0OmkgjJ09gYzGM9rwaIXzoMVOdX67rnrmy9mGuCQq8pdgnPb4ma+5f+X5CllTkDtQYaWIeRgZFfrQYT/zwB2VJmifa73aBpDpYfLO3t3xMV4njpl+XfPAFWL35GMi4B52IuqfAF8aHyUQogo3cm5PuCFUaOxcBJtbCmWqflQsXUWsVTQWztmi4dzfoerz+ME7OtbkFQWsM= ceph-c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:02:55.283 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-13T15:02:55.488 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:55 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/266840789' entity='client.admin' 2024-02-13T15:02:58.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:58.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:02:58.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:58.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:58.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:02:58.432 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-13T15:02:58.432 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-13T15:02:59.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:58 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2615249066' entity='client.admin' 2024-02-13T15:02:59.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:58 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:02:59.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:02:58 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:00.907 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-02-13T15:03:00.907 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:03:00.907 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-02-13T15:03:00.926 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:03:00.926 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:03:00.984 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-02-13T15:03:00.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch host add smithi098 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:01.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:03.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:03 smithi002 ceph-mon[24754]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:03.883 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-02-13T15:03:04.505 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-13T15:03:05.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: Deploying cephadm binary to smithi098 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: Added host smithi098 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:05.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:04 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:06.370 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:03:06.370 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-02-13T15:03:06.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:06 smithi002 ceph-mon[24754]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:06.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:06.977 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-13T15:03:06.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-13T15:03:07.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:07 smithi002 ceph-mon[24754]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:03:08.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:08 smithi002 ceph-mon[24754]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:08.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:08.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:08.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:03:08.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:09.775 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-02-13T15:03:10.080 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:09 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3662644298' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-13T15:03:10.367 INFO:tasks.cephadm:Adding mon.a on smithi002 2024-02-13T15:03:10.367 INFO:tasks.cephadm:Adding mon.c on smithi002 2024-02-13T15:03:10.367 INFO:tasks.cephadm:Adding mon.b on smithi098 2024-02-13T15:03:10.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi098:172.21.15.98=b' 2024-02-13T15:03:11.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:10 smithi002 ceph-mon[24754]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:11.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:10 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3662644298' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-13T15:03:11.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:10 smithi002 ceph-mon[24754]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:12.042 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mon update... 2024-02-13T15:03:12.714 DEBUG:teuthology.orchestra.run.smithi002:mon.c> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.c.service 2024-02-13T15:03:12.717 DEBUG:teuthology.orchestra.run.smithi098:mon.b> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.b.service 2024-02-13T15:03:12.720 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T15:03:12.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi098:172.21.15.98=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi098:172.21.15.98=b;count:3 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:13.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:03:13.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:13.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:13.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:13.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:13 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:14.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:14 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:03:14.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:14 smithi002 ceph-mon[24754]: Deploying daemon mon.c on smithi002 2024-02-13T15:03:14.487 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T15:03:14.487 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","modified":"2024-02-13T15:01:56.789019Z","created":"2024-02-13T15:01:56.789019Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T15:03:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-02-13T15:03:15.264 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:15 smithi002 ceph-mon[24754]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:15.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:15 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/4168967638' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T15:03:16.086 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T15:03:16.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: mon.a calling monitor election 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:21.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: mon.c calling monitor election 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: fsmap 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: mgrmap e13: y(active, since 35s) 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: overall HEALTH_OK 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:21.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T15:03:21.381 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","modified":"2024-02-13T15:03:15.734892Z","created":"2024-02-13T15:01:56.789019Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T15:03:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 2 2024-02-13T15:03:22.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:21 smithi002 ceph-mon[24754]: Deploying daemon mon.b on smithi098 2024-02-13T15:03:22.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:21 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/3827893209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T15:03:22.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:22.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:23.083 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T15:03:23.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T15:03:23.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:22 smithi002 ceph-mon[24754]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:22 smithi002 ceph-mon[29232]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:24 smithi002 ceph-mon[24754]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:25.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:24 smithi002 ceph-mon[29232]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T15:03:25.476 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","modified":"2024-02-13T15:03:15.734892Z","created":"2024-02-13T15:01:56.789019Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T15:03:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 2 2024-02-13T15:03:25.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:25 smithi098 systemd[1]: Started Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:03:27.270 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T15:03:27.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T15:03:31.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: mon.a calling monitor election 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: mon.c calling monitor election 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: fsmap 2024-02-13T15:03:31.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: mgrmap e13: y(active, since 45s) 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:31.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: mon.a calling monitor election 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: mon.c calling monitor election 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: fsmap 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: mgrmap e13: y(active, since 45s) 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T15:03:31.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T15:03:31.133 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-02-13T15:03:31.133 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:31.133 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:31.133 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:31.133 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:31.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-02-13T15:03:31.973 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","modified":"2024-02-13T15:03:25.580100Z","created":"2024-02-13T15:01:56.789019Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T15:03:31.974 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 3 2024-02-13T15:03:32.568 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-13T15:03:32.569 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-13T15:03:32.596 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: Reconfiguring mon.a (unknown last config time)... 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:32.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/1085688816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: Reconfiguring mon.a (unknown last config time)... 2024-02-13T15:03:32.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:32.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:32.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:32.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:03:32.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/1085688816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T15:03:32.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:03:32.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: mon.a calling monitor election 2024-02-13T15:03:32.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: mon.c calling monitor election 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: fsmap 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: mgrmap e13: y(active, since 45s) 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T15:03:32.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:32.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Reconfiguring mon.a (unknown last config time)... 2024-02-13T15:03:32.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:32.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:32.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:32.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:03:32.836 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/1085688816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T15:03:33.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mon.b calling monitor election 2024-02-13T15:03:33.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mon.b calling monitor election 2024-02-13T15:03:33.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mon.a calling monitor election 2024-02-13T15:03:33.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mon.c calling monitor election 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: fsmap 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: mgrmap e13: y(active, since 47s) 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: Cluster is now healthy 2024-02-13T15:03:33.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:33 smithi098 ceph-mon[25694]: overall HEALTH_OK 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mon.b calling monitor election 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mon.b calling monitor election 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mon.a calling monitor election 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mon.c calling monitor election 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: fsmap 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: mgrmap e13: y(active, since 47s) 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:03:33.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: Cluster is now healthy 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[24754]: overall HEALTH_OK 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mon.b calling monitor election 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mon.b calling monitor election 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mon.a calling monitor election 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mon.c calling monitor election 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: fsmap 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: mgrmap e13: y(active, since 47s) 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: Cluster is now healthy 2024-02-13T15:03:33.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:33 smithi002 ceph-mon[29232]: overall HEALTH_OK 2024-02-13T15:03:34.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:34.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:34.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:34 smithi098 ceph-mon[25694]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:34.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[24754]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:34.845 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:34 smithi002 ceph-mon[29232]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:03:35.231 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:03:35.231 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-02-13T15:03:35.232 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:03:35.232 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] 2024-02-13T15:03:36.163 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-13T15:03:36.163 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:03:36.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T15:03:36.234 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:03:36.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:03:36.301 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:03:36.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T15:03:36.332 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:03:36.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:03:36.402 INFO:tasks.cephadm:Adding mgr.y on smithi002 2024-02-13T15:03:36.402 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-02-13T15:03:36.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply mgr '2;smithi002=y;smithi098=x' 2024-02-13T15:03:36.422 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[24754]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:36.423 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:36.423 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3734951319' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:36.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[29232]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:36.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:36.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:36 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3734951319' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:36.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:36 smithi098 ceph-mon[25694]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:36.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:36 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:36.466 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:36 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3734951319' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:37.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:37.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:03:37.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:37.627 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:37.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:37.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:37 smithi098 ceph-mon[25694]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[24754]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:37.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:03:37.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:03:37.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:03:37.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:37.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:37 smithi002 ceph-mon[29232]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:03:38.789 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:38.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:38.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:38.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:38.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:38.790 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:38.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:39.159 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mgr update... 2024-02-13T15:03:39.753 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service 2024-02-13T15:03:39.795 INFO:tasks.cephadm:Deploying OSDs... 2024-02-13T15:03:39.796 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:03:39.796 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T15:03:39.815 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T15:03:39.815 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-13 15:03:15.206565577 +0000 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-13 15:01:00.349267355 +0000 2024-02-13T15:03:39.873 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-13 15:01:00.349267355 +0000 2024-02-13T15:03:39.874 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-13 15:01:00.349267355 +0000 2024-02-13T15:03:39.874 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T15:03:39.943 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-13T15:03:39.943 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-13T15:03:39.943 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00011102 s, 4.6 MB/s 2024-02-13T15:03:39.945 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T15:03:40.003 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-02-13T15:03:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T15:03:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:40.060 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-13T15:03:40.060 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:40.060 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:40.061 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-13 15:03:15.206565577 +0000 2024-02-13T15:03:40.061 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-13 15:01:00.700259565 +0000 2024-02-13T15:03:40.061 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-13 15:01:00.700259565 +0000 2024-02-13T15:03:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-13 15:01:00.700259565 +0000 2024-02-13T15:03:40.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T15:03:40.129 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-13T15:03:40.129 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-13T15:03:40.129 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000109632 s, 4.7 MB/s 2024-02-13T15:03:40.130 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T15:03:40.188 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-02-13T15:03:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T15:03:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-13 15:03:15.207565556 +0000 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-13 15:01:01.028252286 +0000 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-13 15:01:01.028252286 +0000 2024-02-13T15:03:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-13 15:01:01.028252286 +0000 2024-02-13T15:03:40.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T15:03:40.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:40.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: Saving service mgr spec with placement smithi002=y;smithi098=x;count:2 2024-02-13T15:03:40.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[24754]: Deploying daemon mgr.x on smithi098 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: Saving service mgr spec with placement smithi002=y;smithi098=x;count:2 2024-02-13T15:03:40.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.311 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:40 smithi002 ceph-mon[29232]: Deploying daemon mgr.x on smithi098 2024-02-13T15:03:40.312 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-13T15:03:40.312 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-13T15:03:40.312 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000103787 s, 4.9 MB/s 2024-02-13T15:03:40.313 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T15:03:40.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi098=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: Saving service mgr spec with placement smithi002=y;smithi098=x;count:2 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:40.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:40 smithi098 ceph-mon[25694]: Deploying daemon mgr.x on smithi098 2024-02-13T15:03:40.368 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-13 15:03:15.207565556 +0000 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-13 15:01:01.358244962 +0000 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-13 15:01:01.358244962 +0000 2024-02-13T15:03:40.423 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-13 15:01:01.358244962 +0000 2024-02-13T15:03:40.424 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T15:03:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-13T15:03:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-13T15:03:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.4703e-05 s, 5.4 MB/s 2024-02-13T15:03:40.490 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T15:03:40.546 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:03:40.546 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T15:03:40.563 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T15:03:40.563 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-02-13T15:03:40.618 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T15:03:40.618 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-02-13 15:03:24.374086406 +0000 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-02-13 15:01:01.439361440 +0000 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-02-13 15:01:01.439361440 +0000 2024-02-13T15:03:40.619 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-02-13 15:01:01.439361440 +0000 2024-02-13T15:03:40.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T15:03:40.686 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-02-13T15:03:40.686 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-02-13T15:03:40.686 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000242642 s, 2.1 MB/s 2024-02-13T15:03:40.687 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T15:03:40.743 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-02-13 15:03:24.374086406 +0000 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-02-13 15:01:01.780356752 +0000 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-02-13 15:01:01.780356752 +0000 2024-02-13T15:03:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-02-13 15:01:01.780356752 +0000 2024-02-13T15:03:40.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T15:03:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-02-13T15:03:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-02-13T15:03:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000182845 s, 2.8 MB/s 2024-02-13T15:03:40.993 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T15:03:41.008 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-02-13 15:03:40.950865915 +0000 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-02-13 15:01:02.116352134 +0000 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-02-13 15:01:02.116352134 +0000 2024-02-13T15:03:41.069 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-02-13 15:01:02.116352134 +0000 2024-02-13T15:03:41.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T15:03:41.163 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-02-13T15:03:41.163 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-02-13T15:03:41.163 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000196079 s, 2.6 MB/s 2024-02-13T15:03:41.164 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T15:03:41.179 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-02-13T15:03:41.236 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T15:03:41.236 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T15:03:41.236 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-13T15:03:41.236 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T15:03:41.236 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T15:03:41.237 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-02-13 15:03:41.122863643 +0000 2024-02-13T15:03:41.237 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-02-13 15:01:02.443347640 +0000 2024-02-13T15:03:41.237 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-02-13 15:01:02.443347640 +0000 2024-02-13T15:03:41.237 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-02-13 15:01:02.443347640 +0000 2024-02-13T15:03:41.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T15:03:41.301 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 systemd[1]: Starting Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:03:41.305 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-02-13T15:03:41.305 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-02-13T15:03:41.305 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000211555 s, 2.4 MB/s 2024-02-13T15:03:41.307 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T15:03:41.363 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2024-02-13T15:03:41.363 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-13T15:03:41.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 podman[26834]: 2024-02-13 15:03:41.307634101 +0000 UTC m=+0.021522068 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-13T15:03:41.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 podman[26834]: 2024-02-13 15:03:41.412781986 +0000 UTC m=+0.126669991 container create 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, version=8, ceph=True, vendor=Red Hat, Inc., release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-13T15:03:42.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 podman[26834]: 2024-02-13 15:03:41.593288076 +0000 UTC m=+0.307176043 container init 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-02-13T15:03:42.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 podman[26834]: 2024-02-13 15:03:41.595872539 +0000 UTC m=+0.309760504 container start 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-02-13T15:03:42.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 ceph-mgr[26885]: -- 172.21.15.98:0/3188899725 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a2d26fe340 con 0x55a2d26d7000 2024-02-13T15:03:42.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 bash[26834]: 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f 2024-02-13T15:03:42.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:41 smithi098 systemd[1]: Started Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:03:42.416 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:42.102+0000 7f9969355000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:03:42.416 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:42.195+0000 7f9969355000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[24754]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[29232]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:43.012 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:42 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:43.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:42.814+0000 7f9969355000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:03:43.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:42.901+0000 7f9969355000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:42.982+0000 7f9969355000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-mon[25694]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:43.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:42 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:43.395 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.128+0000 7f9969355000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:03:43.395 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.206+0000 7f9969355000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:03:43.395 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.311+0000 7f9969355000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:03:43.395 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.392+0000 7f9969355000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00838388 s, 1.3 GB/s 2024-02-13T15:03:43.521 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-13T15:03:43.771 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.552+0000 7f9969355000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:03:43.771 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:43.634+0000 7f9969355000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:03:43.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[29232]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:43.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:43 smithi002 ceph-mon[24754]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:03:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T15:03:44.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:03:44.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:03:44.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:44.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:43 smithi098 ceph-mon[25694]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:44.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:44.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:44.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:44.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.964 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:44.649+0000 7f9969355000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:03:44.964 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:44.776+0000 7f9969355000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:03:44.964 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:44.962+0000 7f9969355000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:03:44.965 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:45.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2024-02-13T15:03:45.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.047+0000 7f9969355000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:03:45.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.181+0000 7f9969355000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:03:45.785 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.494+0000 7f9969355000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:03:45.786 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.712+0000 7f9969355000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:03:45.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:03:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:03:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:03:46.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.797+0000 7f9969355000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:03:46.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:45.964+0000 7f9969355000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:03:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:46 smithi098 ceph-mon[25694]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:47.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:46.869+0000 7f9969355000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:03:47.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:46 smithi002 ceph-mon[24754]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:47.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:46 smithi002 ceph-mon[29232]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:47.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:47.160+0000 7f9969355000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:03:47.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:47.253+0000 7f9969355000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:03:47.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:03:47.399+0000 7f9969355000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:03:47.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: Standby manager daemon x started 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:03:47.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:03:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:03:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:03:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:47 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:03:48.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:03:48.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:03:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:47 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2173761405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:03:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:48 smithi098 ceph-mon[25694]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:48 smithi098 ceph-mon[25694]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-13T15:03:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:48 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[29232]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[29232]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[24754]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[24754]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-13T15:03:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:03:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:49 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]: dispatch 2024-02-13T15:03:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:49 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]': finished 2024-02-13T15:03:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:49 smithi098 ceph-mon[25694]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T15:03:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]: dispatch 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]': finished 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[24754]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]: dispatch 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3102804553' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5284abfb-fd8b-4176-9fda-76eb0cef736d"}]': finished 2024-02-13T15:03:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[29232]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T15:03:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:03:51.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:50 smithi098 ceph-mon[25694]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:51.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:50 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3434065733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:03:51.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:50 smithi002 ceph-mon[24754]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:50 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3434065733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:03:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:50 smithi002 ceph-mon[29232]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:50 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3434065733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:03:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:52 smithi098 ceph-mon[25694]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:53.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:52 smithi002 ceph-mon[24754]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:53.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:52 smithi002 ceph-mon[29232]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:55.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:54 smithi098 ceph-mon[25694]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:55.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:54 smithi002 ceph-mon[24754]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:55.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:54 smithi002 ceph-mon[29232]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:56.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:55 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:03:56.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:55 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:56.076 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:55 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:03:56.076 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:55 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:56.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:55 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:03:56.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:55 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:03:57.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:56 smithi002 ceph-mon[24754]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:57.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:56 smithi002 ceph-mon[24754]: Deploying daemon osd.0 on smithi002 2024-02-13T15:03:57.051 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:56 smithi002 ceph-mon[29232]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:57.051 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:56 smithi002 ceph-mon[29232]: Deploying daemon osd.0 on smithi002 2024-02-13T15:03:57.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:56 smithi098 ceph-mon[25694]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:57.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:56 smithi098 ceph-mon[25694]: Deploying daemon osd.0 on smithi002 2024-02-13T15:03:58.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:03:58 smithi002 ceph-mon[24754]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:58.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:03:58 smithi002 ceph-mon[29232]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:03:59.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:03:58 smithi098 ceph-mon[25694]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[24754]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[29232]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:00.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:00 smithi098 ceph-mon[25694]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:01.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:01.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:01.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:01.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:02 smithi002 ceph-mon[24754]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:02.868 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:02 smithi002 ceph-mon[29232]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:02 smithi098 ceph-mon[25694]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:04.011 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2024-02-13T15:04:04.332 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.332 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.332 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[29232]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:03 smithi002 ceph-mon[24754]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:03 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:03 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:04.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:03 smithi098 ceph-mon[25694]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:04:04.630 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.0.service 2024-02-13T15:04:04.632 INFO:tasks.cephadm:Deploying osd.1 on smithi002 with /dev/vg_nvme/lv_3... 2024-02-13T15:04:04.632 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:05.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[24754]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:05.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-mon[29232]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:05.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:05.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:05 smithi098 ceph-mon[25694]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:05.621 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[32101]: 2024-02-13T15:04:05.316+0000 7f98fdc53700 -1 osd.0 0 waiting for initial osdmap 2024-02-13T15:04:05.621 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[32101]: 2024-02-13T15:04:05.319+0000 7f98f8deb700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[24754]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[24754]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[24754]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[29232]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[29232]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:06.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[29232]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T15:04:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:06 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:06.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:06 smithi098 ceph-mon[25694]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:06.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:06 smithi098 ceph-mon[25694]: from='osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:06.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:06 smithi098 ceph-mon[25694]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T15:04:06.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:06 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:06.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:06 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:07.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[24754]: osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187] boot 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[24754]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[29232]: osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187] boot 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[29232]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T15:04:07.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:07 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:07.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:07 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:04:07.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:07 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:04:07.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:07 smithi098 ceph-mon[25694]: osd.0 [v2:172.21.15.2:6802/3480677187,v1:172.21.15.2:6803/3480677187] boot 2024-02-13T15:04:07.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:07 smithi098 ceph-mon[25694]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T15:04:07.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:07 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00943737 s, 1.1 GB/s 2024-02-13T15:04:07.820 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-13T15:04:08.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:08 smithi002 ceph-mon[24754]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:08.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:08 smithi002 ceph-mon[24754]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T15:04:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:08 smithi002 ceph-mon[29232]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:08 smithi002 ceph-mon[29232]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T15:04:08.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2024-02-13T15:04:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:08 smithi098 ceph-mon[25694]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T15:04:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:08 smithi098 ceph-mon[25694]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi002 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi002 to 16069M 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi002 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:10.465 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi002 to 16069M 2024-02-13T15:04:10.465 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.465 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:10 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi002 2024-02-13T15:04:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi002 to 16069M 2024-02-13T15:04:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:10 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[29232]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[24754]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:11.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:11 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:11 smithi098 ceph-mon[25694]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:11 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:11 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:11 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:12.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:12 smithi098 ceph-mon[25694]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:12.603 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:12 smithi002 ceph-mon[24754]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:12.604 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:12 smithi002 ceph-mon[29232]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]: dispatch 2024-02-13T15:04:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]': finished 2024-02-13T15:04:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T15:04:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:14 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3898065331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]: dispatch 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]': finished 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3898065331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]: dispatch 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3524631273' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0"}]': finished 2024-02-13T15:04:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T15:04:14.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:14.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:14 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3898065331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:16.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:16 smithi002 ceph-mon[24754]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:16.558 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:16 smithi002 ceph-mon[29232]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:16.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:16 smithi098 ceph-mon[25694]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:18.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:18 smithi002 ceph-mon[24754]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:18.563 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:18 smithi002 ceph-mon[29232]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:18 smithi098 ceph-mon[25694]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:19 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:04:20.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:19 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:19 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:04:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:19 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:19 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:04:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:19 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:21.078 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:20 smithi002 ceph-mon[24754]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:21.078 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:20 smithi002 ceph-mon[24754]: Deploying daemon osd.1 on smithi002 2024-02-13T15:04:21.078 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:20 smithi002 ceph-mon[29232]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:21.078 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:20 smithi002 ceph-mon[29232]: Deploying daemon osd.1 on smithi002 2024-02-13T15:04:21.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:20 smithi098 ceph-mon[25694]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:21.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:20 smithi098 ceph-mon[25694]: Deploying daemon osd.1 on smithi002 2024-02-13T15:04:23.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:22 smithi002 ceph-mon[24754]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:23.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:22 smithi002 ceph-mon[29232]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:23.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:22 smithi098 ceph-mon[25694]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:24.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:24 smithi002 ceph-mon[24754]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:24.860 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:24 smithi002 ceph-mon[29232]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:24 smithi098 ceph-mon[25694]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:26.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:26.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:26 smithi002 ceph-mon[24754]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:26 smithi098 ceph-mon[25694]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:27.088 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:26 smithi002 ceph-mon[29232]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:28.180 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 1 on host 'smithi002' 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:28.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:28.540 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:28 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:28.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:28.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:28.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:28 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:28.862 DEBUG:teuthology.orchestra.run.smithi002:osd.1> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.1.service 2024-02-13T15:04:28.864 INFO:tasks.cephadm:Deploying osd.2 on smithi002 with /dev/vg_nvme/lv_2... 2024-02-13T15:04:28.864 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-13T15:04:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:29 smithi098 ceph-mon[25694]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:04:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:29 smithi098 ceph-mon[25694]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T15:04:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:29 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:29 smithi098 ceph-mon[25694]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:30.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[29232]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:04:30.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[29232]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T15:04:30.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:30.122 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[29232]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:30.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[24754]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:04:30.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[24754]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T15:04:30.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:30.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-mon[24754]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:30.122 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[35060]: 2024-02-13T15:04:29.831+0000 7f7fc8b48700 -1 osd.1 0 waiting for initial osdmap 2024-02-13T15:04:30.122 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[35060]: 2024-02-13T15:04:29.834+0000 7f7fc3ce0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:04:31.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[24754]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:31.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[24754]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:31.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[24754]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T15:04:31.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[29232]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[29232]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[29232]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:30 smithi098 ceph-mon[25694]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:30 smithi098 ceph-mon[25694]: from='osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:30 smithi098 ceph-mon[25694]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T15:04:31.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[24754]: osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471] boot 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[24754]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:04:31.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:04:31.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[29232]: osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471] boot 2024-02-13T15:04:31.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[29232]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T15:04:31.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:31 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00954204 s, 1.1 GB/s 2024-02-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-13T15:04:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:31 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:04:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:31 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:04:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:31 smithi098 ceph-mon[25694]: osd.1 [v2:172.21.15.2:6810/2108456471,v1:172.21.15.2:6811/2108456471] boot 2024-02-13T15:04:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:31 smithi098 ceph-mon[25694]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T15:04:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:31 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:04:32.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2024-02-13T15:04:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:32 smithi002 ceph-mon[24754]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:32 smithi002 ceph-mon[24754]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T15:04:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:32 smithi002 ceph-mon[29232]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:32 smithi002 ceph-mon[29232]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T15:04:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:32 smithi098 ceph-mon[25694]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-13T15:04:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:32 smithi098 ceph-mon[25694]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:35.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:04:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi002 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi002 to 8034M 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi002 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi002 to 8034M 2024-02-13T15:04:35.936 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:35.937 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:35.937 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:35 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:35 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi002 2024-02-13T15:04:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:35 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi002 to 8034M 2024-02-13T15:04:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:35 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:04:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:35 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:04:36.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:35 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:36.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:36 smithi002 ceph-mon[24754]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:36.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:36 smithi002 ceph-mon[24754]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:36.965 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:36 smithi002 ceph-mon[29232]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:36.965 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:36 smithi002 ceph-mon[29232]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:37.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:36 smithi098 ceph-mon[25694]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:37.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:36 smithi098 ceph-mon[25694]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1642042021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]': finished 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[29232]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1642042021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]': finished 2024-02-13T15:04:38.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[24754]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T15:04:38.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:37 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1642042021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:37 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]: dispatch 2024-02-13T15:04:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:37 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "035fa366-8803-4c89-a15c-3b94d2275732"}]': finished 2024-02-13T15:04:38.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:37 smithi098 ceph-mon[25694]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T15:04:38.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:38 smithi002 ceph-mon[29232]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:38 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3286705838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:38 smithi002 ceph-mon[24754]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:38 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3286705838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:38 smithi098 ceph-mon[25694]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:38 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3286705838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:04:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:40 smithi002 ceph-mon[24754]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:41.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:40 smithi002 ceph-mon[29232]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:41.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:40 smithi098 ceph-mon[25694]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:43.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:42 smithi002 ceph-mon[24754]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:43.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:42 smithi002 ceph-mon[29232]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:42 smithi098 ceph-mon[25694]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:45.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[24754]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:45.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:04:45.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:45.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[24754]: Deploying daemon osd.2 on smithi002 2024-02-13T15:04:45.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[29232]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:45.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:04:45.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:45.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:44 smithi002 ceph-mon[29232]: Deploying daemon osd.2 on smithi002 2024-02-13T15:04:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:44 smithi098 ceph-mon[25694]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:04:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:44 smithi098 ceph-mon[25694]: Deploying daemon osd.2 on smithi002 2024-02-13T15:04:46.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:04:46.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:04:46.122 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:04:46.122 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:04:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:04:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:04:47.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:46 smithi002 ceph-mon[24754]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:47.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:46 smithi002 ceph-mon[29232]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:47.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:46 smithi098 ceph-mon[25694]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:48 smithi002 ceph-mon[24754]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:48 smithi002 ceph-mon[29232]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:49.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:48 smithi098 ceph-mon[25694]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:50.485 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[24754]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:50.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:50.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:50.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:50 smithi002 ceph-mon[29232]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:50.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:50 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:50.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:50 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:04:50.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:50 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:04:50.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:50 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:04:50.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:50 smithi098 ceph-mon[25694]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:51.978 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 2 on host 'smithi002' 2024-02-13T15:04:52.968 DEBUG:teuthology.orchestra.run.smithi002:osd.2> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.2.service 2024-02-13T15:04:52.971 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_1... 2024-02-13T15:04:52.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-13T15:04:53.214 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[29232]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:53.214 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:53.214 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[29232]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:53.214 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[29232]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:53.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[24754]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:53.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:53.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[24754]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:53.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:52 smithi002 ceph-mon[24754]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:52 smithi098 ceph-mon[25694]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:52 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:52 smithi098 ceph-mon[25694]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:52 smithi098 ceph-mon[25694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.290 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:54.290 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[37977]: 2024-02-13T15:04:53.987+0000 7f73550b5700 -1 osd.2 0 waiting for initial osdmap 2024-02-13T15:04:54.290 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[37977]: 2024-02-13T15:04:53.989+0000 7f734da48700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: from='osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:04:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:53 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-13T15:04:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-13T15:04:55.051 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-13T15:04:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-13T15:04:55.051 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108811 s, 964 MB/s 2024-02-13T15:04:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-13T15:04:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:54 smithi098 ceph-mon[25694]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:54 smithi098 ceph-mon[25694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:54 smithi098 ceph-mon[25694]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T15:04:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:54 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:54 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[24754]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[24754]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[24754]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[29232]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[29232]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:04:55.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[29232]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T15:04:55.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:54 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:55.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2024-02-13T15:04:56.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:04:56.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:04:56.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325] boot 2024-02-13T15:04:56.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325] boot 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:56.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:55 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: osd.2 [v2:172.21.15.2:6818/1651599325,v1:172.21.15.2:6819/1651599325] boot 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:04:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:55 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[29232]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[29232]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[24754]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[24754]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T15:04:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:56 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T15:04:57.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:56 smithi098 ceph-mon[25694]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-13T15:04:57.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:56 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T15:04:57.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:56 smithi098 ceph-mon[25694]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T15:04:57.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:56 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T15:04:57.518 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39217]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:04:57.518 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39217]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:57.518 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39217]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:57.867 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39226]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:04:57.868 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39226]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:57.868 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39226]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:58.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T15:04:58.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[24754]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T15:04:58.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:58.127 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39249]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:04:58.127 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39249]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:58.127 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:57 smithi002 sudo[39249]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T15:04:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[29232]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T15:04:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:58 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:58.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:58 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T15:04:58.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:58 smithi098 ceph-mon[25694]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T15:04:58.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:58 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:04:58.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39288]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:04:58.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39288]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:58.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39288]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:58.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:04:58.663 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:58.663 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39306]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:58.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39309]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:04:58.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39309]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:58.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39309]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:59.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39312]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:04:59.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39312]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:59.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:58 smithi002 sudo[39312]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:59.444 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[29232]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:04:59.444 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[29232]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T15:04:59.444 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[29232]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-13T15:04:59.444 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39320]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39320]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39320]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[24754]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[24754]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[24754]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-13T15:04:59.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:04:59 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:04:59.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 ceph-mon[25694]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:04:59.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 ceph-mon[25694]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T15:04:59.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 ceph-mon[25694]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-13T15:04:59.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:04:59.710 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39344]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:04:59.710 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39344]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:59.710 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39344]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:04:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:04:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:04:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:04:59 smithi002 sudo[39395]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:05:00.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 sudo[27223]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T15:05:00.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 sudo[27223]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:05:00.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:04:59 smithi098 sudo[27223]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi002 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi002 to 5356M 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi002 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:00 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi002 to 5356M 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 sudo[27226]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 sudo[27226]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 sudo[27226]: pam_unix(sudo:session): session closed for user root 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi002 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:00.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:00 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi002 to 5356M 2024-02-13T15:05:01.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:02 smithi098 ceph-mon[25694]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:02 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]: dispatch 2024-02-13T15:05:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:02 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]': finished 2024-02-13T15:05:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:02 smithi098 ceph-mon[25694]: osdmap e22: 4 total, 3 up, 4 in 2024-02-13T15:05:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:02 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:02.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[24754]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:02.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]: dispatch 2024-02-13T15:05:02.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]': finished 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[24754]: osdmap e22: 4 total, 3 up, 4 in 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[29232]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]: dispatch 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3973769065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72826f52-ca60-4249-8519-a0af528d0cc7"}]': finished 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[29232]: osdmap e22: 4 total, 3 up, 4 in 2024-02-13T15:05:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:02 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:03.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:03 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2976261424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:03.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:03 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2976261424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:03.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:03 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2976261424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:04 smithi098 ceph-mon[25694]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:04.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:04 smithi002 ceph-mon[24754]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:04.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:04 smithi002 ceph-mon[29232]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:06.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:06 smithi002 ceph-mon[24754]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:06.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:06 smithi002 ceph-mon[29232]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:06.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:06 smithi098 ceph-mon[25694]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:08.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[24754]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:08.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:05:08.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:08.564 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[29232]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:08.565 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:05:08.565 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:08 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:08 smithi098 ceph-mon[25694]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:08 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:05:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:08 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:09 smithi002 ceph-mon[24754]: Deploying daemon osd.3 on smithi002 2024-02-13T15:05:09.572 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:09 smithi002 ceph-mon[29232]: Deploying daemon osd.3 on smithi002 2024-02-13T15:05:09.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:09 smithi098 ceph-mon[25694]: Deploying daemon osd.3 on smithi002 2024-02-13T15:05:10.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:10 smithi002 ceph-mon[24754]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:10.562 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:10 smithi002 ceph-mon[29232]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:10 smithi098 ceph-mon[25694]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:12.494 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:12 smithi002 ceph-mon[24754]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:12.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:12 smithi002 ceph-mon[29232]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:12.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:12 smithi098 ceph-mon[25694]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[24754]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[29232]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:14 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:14.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:14 smithi098 ceph-mon[25694]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:14.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:14 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:14.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:14 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:14.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:14 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:14.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:14 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:16.470 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:16 smithi002 ceph-mon[29232]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:16 smithi002 ceph-mon[24754]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:16.513 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2024-02-13T15:05:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:16 smithi098 ceph-mon[25694]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:17.196 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.3.service 2024-02-13T15:05:17.198 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_4... 2024-02-13T15:05:17.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-13T15:05:17.433 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[24754]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[24754]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:17.434 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[29232]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.434 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[29232]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.434 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:17 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:17.514 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:17 smithi098 ceph-mon[25694]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.514 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:17 smithi098 ceph-mon[25694]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:05:17.514 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:17 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: osdmap e23: 4 total, 3 up, 4 in 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[24754]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: osdmap e23: 4 total, 3 up, 4 in 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:18.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-mon[29232]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:18.628 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[40964]: 2024-02-13T15:05:18.365+0000 7f0759c83700 -1 osd.3 0 waiting for initial osdmap 2024-02-13T15:05:18.629 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:05:18 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[40964]: 2024-02-13T15:05:18.368+0000 7f075661e700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:05:18.721 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: osdmap e23: 4 total, 3 up, 4 in 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: from='osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:18.722 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:18 smithi098 ceph-mon[25694]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113393 s, 925 MB/s 2024-02-13T15:05:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[29232]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[29232]: osdmap e24: 4 total, 3 up, 4 in 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[24754]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[24754]: osdmap e24: 4 total, 3 up, 4 in 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:19 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:19 smithi098 ceph-mon[25694]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-13T15:05:19.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:19 smithi098 ceph-mon[25694]: osdmap e24: 4 total, 3 up, 4 in 2024-02-13T15:05:19.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:19 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.669 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:19 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:19.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577] boot 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: osdmap e25: 4 total, 4 up, 4 in 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:20.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:05:20.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:05:20.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:20.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577] boot 2024-02-13T15:05:20.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: osdmap e25: 4 total, 4 up, 4 in 2024-02-13T15:05:20.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: osd.3 [v2:172.21.15.2:6826/55264577,v1:172.21.15.2:6827/55264577] boot 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: osdmap e25: 4 total, 4 up, 4 in 2024-02-13T15:05:20.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:20 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:05:21.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: osdmap e26: 4 total, 4 up, 4 in 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi002 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi002 to 4017M 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:21.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:21 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: osdmap e26: 4 total, 4 up, 4 in 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi002 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi002 to 4017M 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: osdmap e26: 4 total, 4 up, 4 in 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi002 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi002 to 4017M 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:21 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:22.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[24754]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[24754]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[29232]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[29232]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:22.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:22 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:22.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:22 smithi098 ceph-mon[25694]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:22.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:22 smithi098 ceph-mon[25694]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:22.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:22 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:22.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:22 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:22.785 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:22 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/3043012231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]': finished 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: osdmap e27: 5 total, 4 up, 5 in 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:24.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:24 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/891448666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/3043012231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]': finished 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: osdmap e27: 5 total, 4 up, 5 in 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/891448666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/3043012231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]: dispatch 2024-02-13T15:05:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98ee6493-300c-4c52-8d2a-02f4ecd4f9bc"}]': finished 2024-02-13T15:05:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: osdmap e27: 5 total, 4 up, 5 in 2024-02-13T15:05:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:24 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/891448666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:26.635 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:26 smithi098 ceph-mon[25694]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:26.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:26 smithi002 ceph-mon[24754]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:26.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:26 smithi002 ceph-mon[29232]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:28.633 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:28 smithi098 ceph-mon[25694]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:28.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:28 smithi002 ceph-mon[24754]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:28.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:28 smithi002 ceph-mon[29232]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:30.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:30 smithi098 ceph-mon[25694]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:30.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:05:30.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:30.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:30 smithi098 ceph-mon[25694]: Deploying daemon osd.4 on smithi098 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[24754]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[24754]: Deploying daemon osd.4 on smithi098 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[29232]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:30 smithi002 ceph-mon[29232]: Deploying daemon osd.4 on smithi098 2024-02-13T15:05:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:32 smithi098 ceph-mon[25694]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:32.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:32 smithi002 ceph-mon[24754]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:32 smithi002 ceph-mon[29232]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:34.408 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:34 smithi098 ceph-mon[25694]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:34.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:34.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:34.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:34.409 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:34 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[24754]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[29232]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:34.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:34.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:34.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:34 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:36.552 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:36 smithi098 ceph-mon[25694]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:36 smithi002 ceph-mon[24754]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:36.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:36 smithi002 ceph-mon[29232]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:37.480 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-02-13T15:05:37.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:37.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:37 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:38.068 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.4.service 2024-02-13T15:05:38.070 INFO:tasks.cephadm:Deploying osd.5 on smithi098 with /dev/vg_nvme/lv_3... 2024-02-13T15:05:38.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:38.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:05:38.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:38.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:38.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:38.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:37 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:39.150 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:39.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:05:39.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: osdmap e28: 5 total, 4 up, 5 in 2024-02-13T15:05:39.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:39.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:39.151 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-mon[25694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:39.151 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[29067]: 2024-02-13T15:05:38.977+0000 7f52d98f7700 -1 osd.4 0 waiting for initial osdmap 2024-02-13T15:05:39.151 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:05:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[29067]: 2024-02-13T15:05:38.981+0000 7f52d4a8f700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: osdmap e28: 5 total, 4 up, 5 in 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[24754]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:39.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:05:39.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: osdmap e28: 5 total, 4 up, 5 in 2024-02-13T15:05:39.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:39.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: from='osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:39.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:38 smithi002 ceph-mon[29232]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:05:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:05:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: osdmap e29: 5 total, 4 up, 5 in 2024-02-13T15:05:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878] boot 2024-02-13T15:05:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: osdmap e30: 5 total, 5 up, 5 in 2024-02-13T15:05:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:39 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: osdmap e29: 5 total, 4 up, 5 in 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878] boot 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: osdmap e30: 5 total, 5 up, 5 in 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: osdmap e29: 5 total, 4 up, 5 in 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: osd.4 [v2:172.21.15.98:6800/2220714878,v1:172.21.15.98:6801/2220714878] boot 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: osdmap e30: 5 total, 5 up, 5 in 2024-02-13T15:05:40.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:39 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:05:40.905 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-13T15:05:40.905 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-13T15:05:40.905 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-02-13T15:05:40.905 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-02-13T15:05:40.906 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123134 s, 852 MB/s 2024-02-13T15:05:40.906 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-02-13T15:05:41.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:40 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:05:41.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:40 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:05:41.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:40 smithi098 ceph-mon[25694]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:41.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:40 smithi098 ceph-mon[25694]: osdmap e31: 5 total, 5 up, 5 in 2024-02-13T15:05:41.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:05:41.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:05:41.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[24754]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:41.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[24754]: osdmap e31: 5 total, 5 up, 5 in 2024-02-13T15:05:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:05:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:05:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[29232]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-13T15:05:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:40 smithi002 ceph-mon[29232]: osdmap e31: 5 total, 5 up, 5 in 2024-02-13T15:05:41.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-02-13T15:05:42.304 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:42 smithi098 ceph-mon[25694]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:42.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:42 smithi098 ceph-mon[25694]: osdmap e32: 5 total, 5 up, 5 in 2024-02-13T15:05:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:42 smithi002 ceph-mon[24754]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:42.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:42 smithi002 ceph-mon[24754]: osdmap e32: 5 total, 5 up, 5 in 2024-02-13T15:05:42.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:42 smithi002 ceph-mon[29232]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:42.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:42 smithi002 ceph-mon[29232]: osdmap e32: 5 total, 5 up, 5 in 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: osdmap e33: 5 total, 5 up, 5 in 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi098 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi098 to 17088M 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.427 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: osdmap e33: 5 total, 5 up, 5 in 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi098 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi098 to 17088M 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: osdmap e33: 5 total, 5 up, 5 in 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi098 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:05:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi098 to 17088M 2024-02-13T15:05:43.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:43.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:44.447 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-13T15:05:44.448 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: osdmap e34: 5 total, 5 up, 5 in 2024-02-13T15:05:44.448 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:44.448 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:44.448 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:44.448 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:44 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: osdmap e34: 5 total, 5 up, 5 in 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:44.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: osdmap e34: 5 total, 5 up, 5 in 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:05:44.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:44 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:45.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:05:45.497 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:05:45.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:05:45.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:05:45.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:05:45 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:05:45.180+0000 7f84ac62a700 -1 calc_pg_upmaps abort due to max <= 0 2024-02-13T15:05:45.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:05:45.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:05:46.528 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-02-13T15:05:46.528 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/710906626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.528 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]': finished 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: osdmap e35: 6 total, 5 up, 6 in 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/710906626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]': finished 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: osdmap e35: 6 total, 5 up, 6 in 2024-02-13T15:05:46.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:05:46.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-02-13T15:05:46.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/710906626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]: dispatch 2024-02-13T15:05:46.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f9c6145-bc4d-4870-a11c-95ccb18eb8e3"}]': finished 2024-02-13T15:05:46.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: osdmap e35: 6 total, 5 up, 6 in 2024-02-13T15:05:46.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:05:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:47 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/387690110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:47.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:47 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/387690110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:47 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/387690110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:05:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:48 smithi098 ceph-mon[25694]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:05:48.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:48 smithi002 ceph-mon[24754]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:05:48.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:48 smithi002 ceph-mon[29232]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:05:50.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:50 smithi098 ceph-mon[25694]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:50.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:50 smithi002 ceph-mon[24754]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:50.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:50 smithi002 ceph-mon[29232]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:52.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:52 smithi098 ceph-mon[25694]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:52.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:52 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:05:52.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:52 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:52.270 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:52 smithi098 ceph-mon[25694]: Deploying daemon osd.5 on smithi098 2024-02-13T15:05:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[24754]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[24754]: Deploying daemon osd.5 on smithi098 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[29232]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:52 smithi002 ceph-mon[29232]: Deploying daemon osd.5 on smithi098 2024-02-13T15:05:54.530 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:54 smithi098 ceph-mon[25694]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:54 smithi002 ceph-mon[24754]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:54 smithi002 ceph-mon[29232]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:56.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:56 smithi098 ceph-mon[25694]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:56 smithi002 ceph-mon[24754]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:56 smithi002 ceph-mon[29232]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:58.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:57 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:58.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:57 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:05:58.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:57 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:05:58.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:57 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:05:58.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:57 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:05:59.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:05:58 smithi098 ceph-mon[25694]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:59.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:05:58 smithi002 ceph-mon[24754]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:05:59.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:05:58 smithi002 ceph-mon[29232]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:00.230 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 5 on host 'smithi098' 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:00.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:00.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:00.851 DEBUG:teuthology.orchestra.run.smithi098:osd.5> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.5.service 2024-02-13T15:06:00.853 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-02-13T15:06:00.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-13T15:06:00.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:06:00.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:00.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:00.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:00.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:02.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-mon[25694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:06:02.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-mon[25694]: osdmap e36: 6 total, 5 up, 6 in 2024-02-13T15:06:02.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:02.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-mon[25694]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:02.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-mon[25694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:02.083 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[32007]: 2024-02-13T15:06:01.895+0000 7f3f13dc3700 -1 osd.5 0 waiting for initial osdmap 2024-02-13T15:06:02.083 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:06:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[32007]: 2024-02-13T15:06:01.900+0000 7f3f0c756700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[29232]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[29232]: osdmap e36: 6 total, 5 up, 6 in 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[29232]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[29232]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[24754]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[24754]: osdmap e36: 6 total, 5 up, 6 in 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[24754]: from='osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:02.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:01 smithi002 ceph-mon[24754]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:03.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:02 smithi098 ceph-mon[25694]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:03.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:02 smithi098 ceph-mon[25694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:03.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:02 smithi098 ceph-mon[25694]: osdmap e37: 6 total, 5 up, 6 in 2024-02-13T15:06:03.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:02 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.077 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:02 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.108 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-13T15:06:03.108 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-13T15:06:03.108 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-02-13T15:06:03.108 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-02-13T15:06:03.109 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124848 s, 840 MB/s 2024-02-13T15:06:03.109 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-02-13T15:06:03.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[24754]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[24754]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[24754]: osdmap e37: 6 total, 5 up, 6 in 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[29232]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[29232]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[29232]: osdmap e37: 6 total, 5 up, 6 in 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:02 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:03.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-02-13T15:06:04.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:03 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:06:04.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:03 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:06:04.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:03 smithi098 ceph-mon[25694]: osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659] boot 2024-02-13T15:06:04.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:03 smithi098 ceph-mon[25694]: osdmap e38: 6 total, 6 up, 6 in 2024-02-13T15:06:04.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:03 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:04.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[24754]: osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659] boot 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[24754]: osdmap e38: 6 total, 6 up, 6 in 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[29232]: osd.5 [v2:172.21.15.98:6808/2911586659,v1:172.21.15.98:6809/2911586659] boot 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[29232]: osdmap e38: 6 total, 6 up, 6 in 2024-02-13T15:06:04.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:03 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:06:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:04 smithi098 ceph-mon[25694]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:04 smithi098 ceph-mon[25694]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-13T15:06:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:04 smithi098 ceph-mon[25694]: osdmap e39: 6 total, 6 up, 6 in 2024-02-13T15:06:05.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[24754]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:05.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[24754]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-13T15:06:05.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[24754]: osdmap e39: 6 total, 6 up, 6 in 2024-02-13T15:06:05.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[29232]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:05.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[29232]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-13T15:06:05.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:04 smithi002 ceph-mon[29232]: osdmap e39: 6 total, 6 up, 6 in 2024-02-13T15:06:06.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: osdmap e40: 6 total, 6 up, 6 in 2024-02-13T15:06:06.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.172 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:05 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: osdmap e40: 6 total, 6 up, 6 in 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: osdmap e40: 6 total, 6 up, 6 in 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:06.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:05 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi098 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi098 to 8544M 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: osdmap e41: 6 total, 6 up, 6 in 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:07.239 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:06 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi098 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi098 to 8544M 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: osdmap e41: 6 total, 6 up, 6 in 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:07.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi098 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi098 to 8544M 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: osdmap e41: 6 total, 6 up, 6 in 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:07.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:06 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:08.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:07 smithi098 ceph-mon[25694]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:08.212 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:07 smithi098 ceph-mon[25694]: osdmap e42: 6 total, 6 up, 6 in 2024-02-13T15:06:08.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:07 smithi002 ceph-mon[24754]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:08.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:07 smithi002 ceph-mon[24754]: osdmap e42: 6 total, 6 up, 6 in 2024-02-13T15:06:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:07 smithi002 ceph-mon[29232]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:07 smithi002 ceph-mon[29232]: osdmap e42: 6 total, 6 up, 6 in 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/3175552547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]': finished 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: osdmap e43: 7 total, 6 up, 7 in 2024-02-13T15:06:09.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:08 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/3175552547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]': finished 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: osdmap e43: 7 total, 6 up, 7 in 2024-02-13T15:06:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/3175552547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]: dispatch 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4aec8aee-aec6-4823-9920-d27608a3519e"}]': finished 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: osdmap e43: 7 total, 6 up, 7 in 2024-02-13T15:06:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:08 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:10 smithi098 ceph-mon[25694]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-13T15:06:10.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:10 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/2915612712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:10.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:10 smithi098 ceph-mon[25694]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-13T15:06:10.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:10 smithi098 ceph-mon[25694]: Cluster is now healthy 2024-02-13T15:06:10.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[24754]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-13T15:06:10.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/2915612712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:10.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[24754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-13T15:06:10.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[24754]: Cluster is now healthy 2024-02-13T15:06:10.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[29232]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-13T15:06:10.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/2915612712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:10.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[29232]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-13T15:06:10.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:10 smithi002 ceph-mon[29232]: Cluster is now healthy 2024-02-13T15:06:12.476 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:12 smithi098 ceph-mon[25694]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:12.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:12 smithi002 ceph-mon[24754]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:12.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:12 smithi002 ceph-mon[29232]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:14.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:14 smithi098 ceph-mon[25694]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-13T15:06:14.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:14 smithi002 ceph-mon[24754]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-13T15:06:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:14 smithi002 ceph-mon[29232]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-02-13T15:06:15.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:15 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:06:15.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:15 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:15.343 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:15 smithi098 ceph-mon[25694]: Deploying daemon osd.6 on smithi098 2024-02-13T15:06:15.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:06:15.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:15.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[24754]: Deploying daemon osd.6 on smithi098 2024-02-13T15:06:15.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:06:15.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:15.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:15 smithi002 ceph-mon[29232]: Deploying daemon osd.6 on smithi098 2024-02-13T15:06:16.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:16 smithi098 ceph-mon[25694]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-13T15:06:16.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:16 smithi002 ceph-mon[24754]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-13T15:06:16.528 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:16 smithi002 ceph-mon[29232]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-13T15:06:18.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:18 smithi098 ceph-mon[25694]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-13T15:06:18.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:18 smithi002 ceph-mon[24754]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-13T15:06:18.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:18 smithi002 ceph-mon[29232]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-13T15:06:20.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:20 smithi098 ceph-mon[25694]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:20.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:20 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:20.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:20 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:20.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:20 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:20.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:20 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:20.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[24754]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[29232]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:20.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:20.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:20.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:20.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:20 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:22.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:22 smithi098 ceph-mon[25694]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:22.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:22 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:22 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[24754]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:22.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[29232]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:22.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:22 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:22.947 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-02-13T15:06:23.533 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.6.service 2024-02-13T15:06:23.536 INFO:tasks.cephadm:Deploying osd.7 on smithi098 with /dev/vg_nvme/lv_1... 2024-02-13T15:06:23.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-13T15:06:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.818 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.819 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:23.819 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:23.819 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:23.819 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:23 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:23.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:23 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: osdmap e44: 7 total, 6 up, 7 in 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[24754]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:06:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: osdmap e44: 7 total, 6 up, 7 in 2024-02-13T15:06:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:24 smithi002 ceph-mon[29232]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: osdmap e44: 7 total, 6 up, 7 in 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: from='osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.889 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-mon[25694]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:24.889 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[34899]: 2024-02-13T15:06:24.598+0000 7f8c861ba700 -1 osd.6 0 waiting for initial osdmap 2024-02-13T15:06:24.889 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:06:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[34899]: 2024-02-13T15:06:24.605+0000 7f8c7eb4d700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: osdmap e45: 7 total, 6 up, 7 in 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038] boot 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: osdmap e46: 7 total, 7 up, 7 in 2024-02-13T15:06:25.779 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:25 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: osdmap e45: 7 total, 6 up, 7 in 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038] boot 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: osdmap e46: 7 total, 7 up, 7 in 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: osdmap e45: 7 total, 6 up, 7 in 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:25.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: osd.6 [v2:172.21.15.98:6816/1973988038,v1:172.21.15.98:6817/1973988038] boot 2024-02-13T15:06:25.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: osdmap e46: 7 total, 7 up, 7 in 2024-02-13T15:06:25.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:25 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[24754]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[24754]: osdmap e47: 7 total, 7 up, 7 in 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[29232]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:26.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:26 smithi002 ceph-mon[29232]: osdmap e47: 7 total, 7 up, 7 in 2024-02-13T15:06:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:26 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:06:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:26 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:06:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:26 smithi098 ceph-mon[25694]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-13T15:06:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:26 smithi098 ceph-mon[25694]: osdmap e47: 7 total, 7 up, 7 in 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123585 s, 848 MB/s 2024-02-13T15:06:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-02-13T15:06:27.756 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi098 2024-02-13T15:06:27.756 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi098 to 5696M 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.757 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:27 smithi098 ceph-mon[25694]: osdmap e48: 7 total, 7 up, 7 in 2024-02-13T15:06:27.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi098 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi098 to 5696M 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[24754]: osdmap e48: 7 total, 7 up, 7 in 2024-02-13T15:06:27.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi098 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi098 to 5696M 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:27.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:27 smithi002 ceph-mon[29232]: osdmap e48: 7 total, 7 up, 7 in 2024-02-13T15:06:28.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:28 smithi002 ceph-mon[24754]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:28.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:28 smithi002 ceph-mon[29232]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:28.997 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:28 smithi098 ceph-mon[25694]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:29 smithi002 ceph-mon[24754]: osdmap e49: 7 total, 7 up, 7 in 2024-02-13T15:06:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:29 smithi002 ceph-mon[29232]: osdmap e49: 7 total, 7 up, 7 in 2024-02-13T15:06:30.051 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:29 smithi098 ceph-mon[25694]: osdmap e49: 7 total, 7 up, 7 in 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: osdmap e50: 7 total, 7 up, 7 in 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:30.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: osdmap e50: 7 total, 7 up, 7 in 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:30.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:30 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: osdmap e50: 7 total, 7 up, 7 in 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T15:06:30.924 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:30 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/2274056860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]': finished 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: osdmap e51: 8 total, 7 up, 8 in 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/1268405883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/2274056860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]': finished 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: osdmap e51: 8 total, 7 up, 8 in 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/1268405883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/2274056860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]: dispatch 2024-02-13T15:06:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b27efae-537e-4f57-a0d7-bc914da043b5"}]': finished 2024-02-13T15:06:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: osdmap e51: 8 total, 7 up, 8 in 2024-02-13T15:06:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/1268405883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T15:06:35.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:34 smithi098 ceph-mon[25694]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-13T15:06:35.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:34 smithi002 ceph-mon[24754]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-13T15:06:35.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:34 smithi002 ceph-mon[29232]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-13T15:06:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:36 smithi098 ceph-mon[25694]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-13T15:06:37.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:36 smithi002 ceph-mon[24754]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-13T15:06:37.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:36 smithi002 ceph-mon[29232]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-13T15:06:38.749 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:38 smithi098 ceph-mon[25694]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:06:38.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:06:38.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:38 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:38.750 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:38 smithi098 ceph-mon[25694]: Deploying daemon osd.7 on smithi098 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[24754]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[24754]: Deploying daemon osd.7 on smithi098 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[29232]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:38 smithi002 ceph-mon[29232]: Deploying daemon osd.7 on smithi098 2024-02-13T15:06:40.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:40 smithi098 ceph-mon[25694]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-13T15:06:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:40 smithi002 ceph-mon[24754]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-13T15:06:41.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:40 smithi002 ceph-mon[29232]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-02-13T15:06:42.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:42 smithi098 ceph-mon[25694]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:43.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:42 smithi002 ceph-mon[24754]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:43.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:42 smithi002 ceph-mon[29232]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:44.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:44.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:44.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:44.204 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:43 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:43 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:45.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:44 smithi098 ceph-mon[25694]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:45.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:44 smithi002 ceph-mon[24754]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:45.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:44 smithi002 ceph-mon[29232]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:46.152 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:06:46.153 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:06:46.153 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:45 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:46.162 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 7 on host 'smithi098' 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:06:46.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:45 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:46.759 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.7.service 2024-02-13T15:06:46.763 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-13T15:06:46.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-13T15:06:46.986 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:46.987 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:46 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:47.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:47.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:47.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:06:47.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:47.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:06:47.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:06:47.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:46 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:06:48.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[29232]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:06:48.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[29232]: osdmap e52: 8 total, 7 up, 8 in 2024-02-13T15:06:48.085 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[29232]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[29232]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[24754]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[24754]: osdmap e52: 8 total, 7 up, 8 in 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[24754]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:47 smithi002 ceph-mon[24754]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-mon[25694]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:06:48.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-mon[25694]: osdmap e52: 8 total, 7 up, 8 in 2024-02-13T15:06:48.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:48.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-mon[25694]: from='osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.217 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-mon[25694]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:06:48.217 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[37833]: 2024-02-13T15:06:47.969+0000 7fbc41838700 -1 osd.7 0 waiting for initial osdmap 2024-02-13T15:06:48.217 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:06:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[37833]: 2024-02-13T15:06:47.977+0000 7fbc3c1cf700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:06:48.720 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: osdmap e53: 8 total, 7 up, 8 in 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2487589882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:49.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: osdmap e53: 8 total, 7 up, 8 in 2024-02-13T15:06:49.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:48 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2487589882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: osdmap e53: 8 total, 7 up, 8 in 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:49.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:48 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2487589882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:49.521 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1707836785,"num_in_osds":8,"osd_in_since":1707836791,"num_remapped_pgs":0} 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: purged_snaps scrub starts 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: purged_snaps scrub ok 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699] boot 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: osdmap e54: 8 total, 8 up, 8 in 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: osdmap e55: 8 total, 8 up, 8 in 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: purged_snaps scrub starts 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: purged_snaps scrub ok 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699] boot 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: osdmap e54: 8 total, 8 up, 8 in 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: osdmap e55: 8 total, 8 up, 8 in 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:49 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: purged_snaps scrub starts 2024-02-13T15:06:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: purged_snaps scrub ok 2024-02-13T15:06:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: osd.7 [v2:172.21.15.98:6824/3790380699,v1:172.21.15.98:6825/3790380699] boot 2024-02-13T15:06:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: osdmap e54: 8 total, 8 up, 8 in 2024-02-13T15:06:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: osdmap e55: 8 total, 8 up, 8 in 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:49 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:06:50.522 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[24754]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[24754]: Detected new or changed devices on smithi098 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi098 to 4272M 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[24754]: osdmap e56: 8 total, 8 up, 8 in 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[29232]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[29232]: Detected new or changed devices on smithi098 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi098 to 4272M 2024-02-13T15:06:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:50 smithi002 ceph-mon[29232]: osdmap e56: 8 total, 8 up, 8 in 2024-02-13T15:06:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:50 smithi098 ceph-mon[25694]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:50 smithi098 ceph-mon[25694]: Detected new or changed devices on smithi098 2024-02-13T15:06:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:50 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi098 to 4272M 2024-02-13T15:06:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:50 smithi098 ceph-mon[25694]: osdmap e56: 8 total, 8 up, 8 in 2024-02-13T15:06:52.462 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[29232]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[29232]: osdmap e57: 8 total, 8 up, 8 in 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2198729277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[24754]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[24754]: osdmap e57: 8 total, 8 up, 8 in 2024-02-13T15:06:53.257 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:52 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2198729277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:53.284 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1707836808,"num_in_osds":8,"osd_in_since":1707836791,"num_remapped_pgs":0} 2024-02-13T15:06:53.285 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T15:06:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:52 smithi098 ceph-mon[25694]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:52 smithi098 ceph-mon[25694]: osdmap e57: 8 total, 8 up, 8 in 2024-02-13T15:06:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:52 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2198729277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T15:06:53.509 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:06:54.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:53 smithi002 ceph-mon[24754]: osdmap e58: 8 total, 8 up, 8 in 2024-02-13T15:06:54.331 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:53 smithi002 ceph-mon[29232]: osdmap e58: 8 total, 8 up, 8 in 2024-02-13T15:06:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:53 smithi098 ceph-mon[25694]: osdmap e58: 8 total, 8 up, 8 in 2024-02-13T15:06:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:54 smithi098 ceph-mon[25694]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:55.364 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:06:55.364 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":58,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","created":"2024-02-13T15:01:59.607220+0000","modified":"2024-02-13T15:06:52.974016+0000","last_up_change":"2024-02-13T15:06:48.959298+0000","last_in_change":"2024-02-13T15:06:31.854208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T15:04:55.478510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5284abfb-fd8b-4176-9fda-76eb0cef736d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6803","nonce":3480677187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6805","nonce":3480677187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6809","nonce":3480677187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6807","nonce":3480677187}]},"public_addr":"172.21.15.2:6803/3480677187","cluster_addr":"172.21.15.2:6805/3480677187","heartbeat_back_addr":"172.21.15.2:6809/3480677187","heartbeat_front_addr":"172.21.15.2:6807/3480677187","state":["exists","up"]},{"osd":1,"uuid":"11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6811","nonce":2108456471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6813","nonce":2108456471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6817","nonce":2108456471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6815","nonce":2108456471}]},"public_addr":"172.21.15.2:6811/2108456471","cluster_addr":"172.21.15.2:6813/2108456471","heartbeat_back_addr":"172.21.15.2:6817/2108456471","heartbeat_front_addr":"172.21.15.2:6815/2108456471","state":["exists","up"]},{"osd":2,"uuid":"035fa366-8803-4c89-a15c-3b94d2275732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6819","nonce":1651599325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6821","nonce":1651599325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6825","nonce":1651599325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6823","nonce":1651599325}]},"public_addr":"172.21.15.2:6819/1651599325","cluster_addr":"172.21.15.2:6821/1651599325","heartbeat_back_addr":"172.21.15.2:6825/1651599325","heartbeat_front_addr":"172.21.15.2:6823/1651599325","state":["exists","up"]},{"osd":3,"uuid":"72826f52-ca60-4249-8519-a0af528d0cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6827","nonce":55264577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6829","nonce":55264577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6833","nonce":55264577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6831","nonce":55264577}]},"public_addr":"172.21.15.2:6827/55264577","cluster_addr":"172.21.15.2:6829/55264577","heartbeat_back_addr":"172.21.15.2:6833/55264577","heartbeat_front_addr":"172.21.15.2:6831/55264577","state":["exists","up"]},{"osd":4,"uuid":"98ee6493-300c-4c52-8d2a-02f4ecd4f9bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6801","nonce":2220714878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6803","nonce":2220714878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6807","nonce":2220714878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6805","nonce":2220714878}]},"public_addr":"172.21.15.98:6801/2220714878","cluster_addr":"172.21.15.98:6803/2220714878","heartbeat_back_addr":"172.21.15.98:6807/2220714878","heartbeat_front_addr":"172.21.15.98:6805/2220714878","state":["exists","up"]},{"osd":5,"uuid":"9f9c6145-bc4d-4870-a11c-95ccb18eb8e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6809","nonce":2911586659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6811","nonce":2911586659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6815","nonce":2911586659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6813","nonce":2911586659}]},"public_addr":"172.21.15.98:6809/2911586659","cluster_addr":"172.21.15.98:6811/2911586659","heartbeat_back_addr":"172.21.15.98:6815/2911586659","heartbeat_front_addr":"172.21.15.98:6813/2911586659","state":["exists","up"]},{"osd":6,"uuid":"4aec8aee-aec6-4823-9920-d27608a3519e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6817","nonce":1973988038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1973988038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1973988038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1973988038}]},"public_addr":"172.21.15.98:6817/1973988038","cluster_addr":"172.21.15.98:6819/1973988038","heartbeat_back_addr":"172.21.15.98:6823/1973988038","heartbeat_front_addr":"172.21.15.98:6821/1973988038","state":["exists","up"]},{"osd":7,"uuid":"8b27efae-537e-4f57-a0d7-bc914da043b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6825","nonce":3790380699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6827","nonce":3790380699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6831","nonce":3790380699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6829","nonce":3790380699}]},"public_addr":"172.21.15.98:6825/3790380699","cluster_addr":"172.21.15.98:6827/3790380699","heartbeat_back_addr":"172.21.15.98:6831/3790380699","heartbeat_front_addr":"172.21.15.98:6829/3790380699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:04.636954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:28.880952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:53.268030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:17.422334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:38.397942+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:00.971584+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:23.674182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:47.038045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/291642484":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/1730337629":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/219490389":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6801/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6800/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:6801/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/2721686623":"2024-02-14T15:02:22.695373+0000","172.21.15.2:6800/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/2705129878":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/760776131":"2024-02-14T15:02:22.695373+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T15:06:55.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:54 smithi002 ceph-mon[24754]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:55.365 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:54 smithi002 ceph-mon[29232]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:06:56.064 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-13T15:04:55.478510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-13T15:06:56.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-13T15:06:56.283 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:06:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:55 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3890427704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:06:56.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:55 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3890427704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:06:56.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:55 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3890427704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:06:57.249 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:56 smithi098 ceph-mon[25694]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:57.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:56 smithi002 ceph-mon[24754]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:57.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:56 smithi002 ceph-mon[29232]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-13T15:06:58.127 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-02-13T15:06:58.775 INFO:tasks.cephadm:Adding prometheus.a on smithi098 2024-02-13T15:06:58.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi098=a' 2024-02-13T15:06:59.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:58 smithi098 ceph-mon[25694]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-13T15:06:59.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:06:58 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3772968753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T15:06:59.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:58 smithi002 ceph-mon[24754]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-13T15:06:59.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:06:58 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3772968753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T15:06:59.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:58 smithi002 ceph-mon[29232]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-13T15:06:59.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:06:58 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3772968753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T15:07:00.708 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled prometheus update... 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T15:07:01.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:00 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' 2024-02-13T15:07:01.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:00 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T15:07:01.286 DEBUG:teuthology.orchestra.run.smithi098:prometheus.a> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-13T15:07:01.288 INFO:tasks.cephadm:Adding node-exporter.a on smithi002 2024-02-13T15:07:01.288 INFO:tasks.cephadm:Adding node-exporter.b on smithi098 2024-02-13T15:07:01.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '2;smithi002=a;smithi098=b' 2024-02-13T15:07:01.909 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ignoring --setuser ceph since I am not root 2024-02-13T15:07:01.909 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ignoring --setgroup ceph since I am not root 2024-02-13T15:07:01.909 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:01 smithi098 ceph-mgr[26885]: -- 172.21.15.98:0/942696810 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55974395c340 con 0x559743935000 2024-02-13T15:07:02.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[24754]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:02.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[24754]: Saving service prometheus spec with placement smithi098=a;count:1 2024-02-13T15:07:02.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[24754]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T15:07:02.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[24754]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-13T15:07:02.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[29232]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[29232]: Saving service prometheus spec with placement smithi098=a;count:1 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[29232]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-mon[29232]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setuser ceph since I am not root 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setgroup ceph since I am not root 2024-02-13T15:07:02.128 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:01 smithi002 ceph-mgr[24990]: -- 172.21.15.2:0/1798201717 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555f8d084340 con 0x555f8d05c800 2024-02-13T15:07:02.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-mon[25694]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:02.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-mon[25694]: Saving service prometheus spec with placement smithi098=a;count:1 2024-02-13T15:07:02.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-mon[25694]: from='mgr.14148 172.21.15.2:0/4121618653' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T15:07:02.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-mon[25694]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-13T15:07:02.540 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:02.262+0000 7f5423dc7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:07:02.540 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:02.352+0000 7f5423dc7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:07:02.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:02.275+0000 7f70ced3a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:07:02.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:02.366+0000 7f70ced3a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:07:03.150 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:02.965+0000 7f5423dc7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:07:03.150 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.057+0000 7f5423dc7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:07:03.150 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.147+0000 7f5423dc7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:07:03.301 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:02.971+0000 7f70ced3a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:07:03.301 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.062+0000 7f70ced3a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:07:03.301 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.147+0000 7f70ced3a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:07:03.301 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.298+0000 7f70ced3a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:07:03.487 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.299+0000 7f5423dc7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:07:03.488 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.378+0000 7f5423dc7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:07:03.488 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.486+0000 7f5423dc7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:07:03.580 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.380+0000 7f70ced3a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:07:03.580 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.492+0000 7f70ced3a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:07:03.814 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.568+0000 7f5423dc7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:07:03.814 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.728+0000 7f5423dc7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:07:03.814 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:03.812+0000 7f5423dc7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:07:03.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.577+0000 7f70ced3a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:07:03.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.743+0000 7f70ced3a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:07:04.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:03.830+0000 7f70ced3a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:07:05.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:04.822+0000 7f5423dc7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:07:05.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:04.950+0000 7f5423dc7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:07:05.208 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:04.877+0000 7f70ced3a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:07:05.208 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.010+0000 7f70ced3a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:07:05.208 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.205+0000 7f70ced3a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:07:05.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.138+0000 7f5423dc7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:07:05.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.224+0000 7f5423dc7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:07:05.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.355+0000 7f5423dc7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:07:05.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.295+0000 7f70ced3a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:07:05.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.434+0000 7f70ced3a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:07:05.966 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.666+0000 7f5423dc7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:07:05.967 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.882+0000 7f5423dc7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:07:05.967 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:05.965+0000 7f5423dc7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:07:06.076 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.758+0000 7f70ced3a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:07:06.077 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:05.986+0000 7f70ced3a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:07:06.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:06.132+0000 7f5423dc7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:07:06.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:06.074+0000 7f70ced3a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:07:06.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:06.249+0000 7f70ced3a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:07:07.311 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:07.022+0000 7f5423dc7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:07:07.311 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:07.309+0000 7f5423dc7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:07:07.504 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:07.196+0000 7f70ced3a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:07:07.562 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:07.401+0000 7f5423dc7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:07:07.562 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:07.546+0000 7f5423dc7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:07:07.562 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: [13/Feb/2024:15:07:07] ENGINE Bus STARTING 2024-02-13T15:07:07.762 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:07.501+0000 7f70ced3a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:07:07.762 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:07.601+0000 7f70ced3a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:07:07.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: Standby manager daemon x restarted 2024-02-13T15:07:07.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: Standby manager daemon x started 2024-02-13T15:07:07.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: Standby manager daemon x restarted 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:07:07.763 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:07:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: Standby manager daemon x restarted 2024-02-13T15:07:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:07:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:07:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:07:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2783625856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: CherryPy Checker: 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: The Application mounted at '' has an empty config. 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: [13/Feb/2024:15:07:07] ENGINE Serving on http://:::9283 2024-02-13T15:07:07.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: [13/Feb/2024:15:07:07] ENGINE Bus STARTED 2024-02-13T15:07:08.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:07.759+0000 7f70ced3a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:07:08.970 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:08] ENGINE Bus STARTING 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[24754]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[24754]: Active manager daemon y restarted 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[24754]: Activating manager daemon y 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[24754]: osdmap e59: 8 total, 8 up, 8 in 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[29232]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[29232]: Active manager daemon y restarted 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[29232]: Activating manager daemon y 2024-02-13T15:07:08.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-mon[29232]: osdmap e59: 8 total, 8 up, 8 in 2024-02-13T15:07:09.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:08 smithi098 ceph-mon[25694]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-13T15:07:09.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:08 smithi098 ceph-mon[25694]: Active manager daemon y restarted 2024-02-13T15:07:09.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:08 smithi098 ceph-mon[25694]: Activating manager daemon y 2024-02-13T15:07:09.028 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:08 smithi098 ceph-mon[25694]: osdmap e59: 8 total, 8 up, 8 in 2024-02-13T15:07:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: CherryPy Checker: 2024-02-13T15:07:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: The Application mounted at '' has an empty config. 2024-02-13T15:07:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:07:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:08] ENGINE Serving on http://:::9283 2024-02-13T15:07:09.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:08 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:08] ENGINE Bus STARTED 2024-02-13T15:07:09.649 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled node-exporter update... 2024-02-13T15:07:09.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:09] ENGINE Bus STARTING 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:07:09] ENGINE Bus STARTED 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: mgrmap e18: y(active, starting, since 0.850839s), standbys: x 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:07:09.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: Manager daemon y is now available 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: mgrmap e18: y(active, starting, since 0.850839s), standbys: x 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:07:09.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: Manager daemon y is now available 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:09.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:09.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:09.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:07:09.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: mgrmap e18: y(active, starting, since 0.850839s), standbys: x 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:07:09.932 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: Manager daemon y is now available 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:09.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:07:09.934 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:07:10.306 DEBUG:teuthology.orchestra.run.smithi002:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-13T15:07:10.309 DEBUG:teuthology.orchestra.run.smithi098:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-13T15:07:10.311 INFO:tasks.cephadm:Adding alertmanager.a on smithi002 2024-02-13T15:07:10.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi002=a' 2024-02-13T15:07:10.973 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: [13/Feb/2024:15:07:09] ENGINE Bus STARTING 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: mgrmap e19: y(active, since 1.86383s), standbys: x 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: Saving service node-exporter spec with placement smithi002=a;smithi098=b;count:2 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: [13/Feb/2024:15:07:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: [13/Feb/2024:15:07:09] ENGINE Bus STARTED 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:10.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: [13/Feb/2024:15:07:09] ENGINE Bus STARTING 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: mgrmap e19: y(active, since 1.86383s), standbys: x 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: Saving service node-exporter spec with placement smithi002=a;smithi098=b;count:2 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: [13/Feb/2024:15:07:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: [13/Feb/2024:15:07:09] ENGINE Bus STARTED 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: [13/Feb/2024:15:07:09] ENGINE Bus STARTING 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: mgrmap e19: y(active, since 1.86383s), standbys: x 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi098=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: Saving service node-exporter spec with placement smithi002=a;smithi098=b;count:2 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: [13/Feb/2024:15:07:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:07:11.019 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: [13/Feb/2024:15:07:09] ENGINE Bus STARTED 2024-02-13T15:07:11.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.898 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:11 smithi098 ceph-mon[25694]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:11.899 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:11 smithi098 ceph-mon[25694]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-13T15:07:11.899 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:11.899 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[24754]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[24754]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[29232]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[29232]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-13T15:07:12.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.336 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled alertmanager update... 2024-02-13T15:07:12.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:12 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:07:12.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:12 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:12.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:12 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.933 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:12 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.955 DEBUG:teuthology.orchestra.run.smithi002:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-13T15:07:12.958 INFO:tasks.cephadm:Adding grafana.a on smithi098 2024-02-13T15:07:12.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi098=a' 2024-02-13T15:07:12.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:07:12.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:12.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:07:12.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:12.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:12.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:12 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:14.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-02-13T15:07:14.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:14.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:14.785 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled grafana update... 2024-02-13T15:07:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:14 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:07:15.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:14 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:15.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:14 smithi098 ceph-mon[25694]: Deploying daemon prometheus.a on smithi098 2024-02-13T15:07:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:07:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[24754]: Deploying daemon prometheus.a on smithi098 2024-02-13T15:07:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:07:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:07:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:14 smithi002 ceph-mon[29232]: Deploying daemon prometheus.a on smithi098 2024-02-13T15:07:15.540 DEBUG:teuthology.orchestra.run.smithi098:grafana.a> sudo journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@grafana.a.service 2024-02-13T15:07:15.543 INFO:tasks.cephadm:Setting up client nodes... 2024-02-13T15:07:15.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[24754]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[24754]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[24754]: Saving service grafana spec with placement smithi098=a;count:1 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[29232]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:15.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[29232]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:15.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[29232]: Saving service grafana spec with placement smithi098=a;count:1 2024-02-13T15:07:15.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:15 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:16.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:15 smithi098 ceph-mon[25694]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:16.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:15 smithi098 ceph-mon[25694]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:07:16.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:15 smithi098 ceph-mon[25694]: Saving service grafana spec with placement smithi098=a;count:1 2024-02-13T15:07:16.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:15 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:17.673 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-02-13T15:07:17.674 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQClhctlZffYJxAAC83FsBt5PkRYEW67Jg8dBA== 2024-02-13T15:07:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:17 smithi098 ceph-mon[25694]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:17 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/840366040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:17 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:17 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[24754]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/840366040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[29232]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/840366040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:17 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:18.524 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-13T15:07:18.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-13T15:07:18.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-13T15:07:18.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T15:07:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:19 smithi098 ceph-mon[25694]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:19 smithi002 ceph-mon[24754]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:20.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:19 smithi002 ceph-mon[29232]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:22.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:21 smithi098 ceph-mon[25694]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:22.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:21 smithi002 ceph-mon[24754]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:22.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:21 smithi002 ceph-mon[29232]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-02-13T15:07:23.165 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCrhctl5qZ+CRAAM8TF+c8q2SItNUmM7VD1dw== 2024-02-13T15:07:23.798 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-02-13T15:07:23.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-13T15:07:23.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-13T15:07:23.841 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-13T15:07:23.841 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-13T15:07:23.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-13T15:07:23.999 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:23 smithi098 ceph-mon[25694]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:23.999 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:23 smithi098 ceph-mon[25694]: from='client.? 172.21.15.98:0/2842991085' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:23.999 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:23 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:23.999 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:23 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:24.068 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[24754]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[24754]: from='client.? 172.21.15.98:0/2842991085' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[29232]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[29232]: from='client.? 172.21.15.98:0/2842991085' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T15:07:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:23 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T15:07:25.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:25 smithi002 ceph-mon[24754]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:25.542 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:25 smithi002 ceph-mon[29232]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:25.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:25 smithi098 ceph-mon[25694]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:26.027 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:07:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:26 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1284662088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T15:07:26.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:26 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1284662088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T15:07:26.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:26 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1284662088' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T15:07:26.786 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"active_gid":14442,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3088002110},{"type":"v1","addr":"172.21.15.2:6801","nonce":3088002110}]},"active_addr":"172.21.15.2:6801/3088002110","active_change":"2024-02-13T15:07:07.774704+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24350,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/","prometheus":"http://172.21.15.2:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1838017178}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2833814139}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2481248937}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3497843875}]}]}} 2024-02-13T15:07:26.790 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-13T15:07:26.790 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-13T15:07:26.790 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T15:07:27.036 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[24754]: Deploying daemon node-exporter.a on smithi002 2024-02-13T15:07:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[24754]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:27.367 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:27.367 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[29232]: Deploying daemon node-exporter.a on smithi002 2024-02-13T15:07:27.367 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:27 smithi002 ceph-mon[29232]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:27.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:27 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:27.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:27 smithi098 ceph-mon[25694]: Deploying daemon node-exporter.a on smithi002 2024-02-13T15:07:27.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:27 smithi098 ceph-mon[25694]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:28.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:07:28.803 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","created":"2024-02-13T15:01:59.607220+0000","modified":"2024-02-13T15:07:07.774392+0000","last_up_change":"2024-02-13T15:06:48.959298+0000","last_in_change":"2024-02-13T15:06:31.854208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T15:04:55.478510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5284abfb-fd8b-4176-9fda-76eb0cef736d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6803","nonce":3480677187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6805","nonce":3480677187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6809","nonce":3480677187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6807","nonce":3480677187}]},"public_addr":"172.21.15.2:6803/3480677187","cluster_addr":"172.21.15.2:6805/3480677187","heartbeat_back_addr":"172.21.15.2:6809/3480677187","heartbeat_front_addr":"172.21.15.2:6807/3480677187","state":["exists","up"]},{"osd":1,"uuid":"11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6811","nonce":2108456471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6813","nonce":2108456471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6817","nonce":2108456471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6815","nonce":2108456471}]},"public_addr":"172.21.15.2:6811/2108456471","cluster_addr":"172.21.15.2:6813/2108456471","heartbeat_back_addr":"172.21.15.2:6817/2108456471","heartbeat_front_addr":"172.21.15.2:6815/2108456471","state":["exists","up"]},{"osd":2,"uuid":"035fa366-8803-4c89-a15c-3b94d2275732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6819","nonce":1651599325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6821","nonce":1651599325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6825","nonce":1651599325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6823","nonce":1651599325}]},"public_addr":"172.21.15.2:6819/1651599325","cluster_addr":"172.21.15.2:6821/1651599325","heartbeat_back_addr":"172.21.15.2:6825/1651599325","heartbeat_front_addr":"172.21.15.2:6823/1651599325","state":["exists","up"]},{"osd":3,"uuid":"72826f52-ca60-4249-8519-a0af528d0cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6827","nonce":55264577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6829","nonce":55264577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6833","nonce":55264577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6831","nonce":55264577}]},"public_addr":"172.21.15.2:6827/55264577","cluster_addr":"172.21.15.2:6829/55264577","heartbeat_back_addr":"172.21.15.2:6833/55264577","heartbeat_front_addr":"172.21.15.2:6831/55264577","state":["exists","up"]},{"osd":4,"uuid":"98ee6493-300c-4c52-8d2a-02f4ecd4f9bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6801","nonce":2220714878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6803","nonce":2220714878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6807","nonce":2220714878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6805","nonce":2220714878}]},"public_addr":"172.21.15.98:6801/2220714878","cluster_addr":"172.21.15.98:6803/2220714878","heartbeat_back_addr":"172.21.15.98:6807/2220714878","heartbeat_front_addr":"172.21.15.98:6805/2220714878","state":["exists","up"]},{"osd":5,"uuid":"9f9c6145-bc4d-4870-a11c-95ccb18eb8e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6809","nonce":2911586659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6811","nonce":2911586659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6815","nonce":2911586659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6813","nonce":2911586659}]},"public_addr":"172.21.15.98:6809/2911586659","cluster_addr":"172.21.15.98:6811/2911586659","heartbeat_back_addr":"172.21.15.98:6815/2911586659","heartbeat_front_addr":"172.21.15.98:6813/2911586659","state":["exists","up"]},{"osd":6,"uuid":"4aec8aee-aec6-4823-9920-d27608a3519e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6817","nonce":1973988038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1973988038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1973988038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1973988038}]},"public_addr":"172.21.15.98:6817/1973988038","cluster_addr":"172.21.15.98:6819/1973988038","heartbeat_back_addr":"172.21.15.98:6823/1973988038","heartbeat_front_addr":"172.21.15.98:6821/1973988038","state":["exists","up"]},{"osd":7,"uuid":"8b27efae-537e-4f57-a0d7-bc914da043b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6825","nonce":3790380699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6827","nonce":3790380699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6831","nonce":3790380699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6829","nonce":3790380699}]},"public_addr":"172.21.15.98:6825/3790380699","cluster_addr":"172.21.15.98:6827/3790380699","heartbeat_back_addr":"172.21.15.98:6831/3790380699","heartbeat_front_addr":"172.21.15.98:6829/3790380699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:04.636954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:28.880952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:53.268030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:17.422334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:38.397942+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:00.971584+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:23.674182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:47.038045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1096752096":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/1844860704":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/604203265":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/291642484":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/1730337629":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/219490389":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6801/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6800/1111087951":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6800/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/3958364651":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6801/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/2721686623":"2024-02-14T15:02:22.695373+0000","172.21.15.2:6801/1111087951":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6800/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/2705129878":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/760776131":"2024-02-14T15:02:22.695373+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T15:07:29.127 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:28 smithi002 bash[44220]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-13T15:07:29.869 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-13T15:07:29.870 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T15:07:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:29 smithi098 ceph-mon[25694]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:29 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:29 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2911179133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[24754]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2911179133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[29232]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:29 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2911179133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:30.591 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:30.965 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:30 smithi002 bash[44220]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-13T15:07:31.297 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 bash[44220]: Writing manifest to image destination 2024-02-13T15:07:31.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 podman[44220]: 2024-02-13 15:07:31.508239931 +0000 UTC m=+3.122233958 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-13T15:07:31.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-mon[24754]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:31.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-mon[29232]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:31.994 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 podman[44220]: 2024-02-13 15:07:31.673985233 +0000 UTC m=+3.287979302 container create 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 podman[44220]: 2024-02-13 15:07:31.89292857 +0000 UTC m=+3.506922604 container init 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 podman[44220]: 2024-02-13 15:07:31.895995421 +0000 UTC m=+3.509989447 container start 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.900Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.900Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T15:07:31.995 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T15:07:31.996 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T15:07:31.996 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T15:07:31.996 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T15:07:31.996 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T15:07:31.996 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T15:07:31.997 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T15:07:31.998 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.902Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[44389]: ts=2024-02-13T15:07:31.903Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T15:07:31.999 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 bash[44220]: 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 2024-02-13T15:07:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:31 smithi098 ceph-mon[25694]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:32.376 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:07:31 smithi002 systemd[1]: Started Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:07:32.773 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:32] "GET /metrics HTTP/1.1" 200 191473 "" "Prometheus/2.33.4" 2024-02-13T15:07:33.058 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:32 smithi098 systemd[1]: Starting Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:07:33.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:07:33.206 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"fsid":"c0fca16a-ca80-11ee-95b9-87774f69a715","created":"2024-02-13T15:01:59.607220+0000","modified":"2024-02-13T15:07:07.774392+0000","last_up_change":"2024-02-13T15:06:48.959298+0000","last_in_change":"2024-02-13T15:06:31.854208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T15:04:55.478510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5284abfb-fd8b-4176-9fda-76eb0cef736d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6803","nonce":3480677187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6805","nonce":3480677187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6809","nonce":3480677187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3480677187},{"type":"v1","addr":"172.21.15.2:6807","nonce":3480677187}]},"public_addr":"172.21.15.2:6803/3480677187","cluster_addr":"172.21.15.2:6805/3480677187","heartbeat_back_addr":"172.21.15.2:6809/3480677187","heartbeat_front_addr":"172.21.15.2:6807/3480677187","state":["exists","up"]},{"osd":1,"uuid":"11e694ea-f4f8-46ba-b1de-a68ad4cf6bb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6811","nonce":2108456471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6813","nonce":2108456471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6817","nonce":2108456471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2108456471},{"type":"v1","addr":"172.21.15.2:6815","nonce":2108456471}]},"public_addr":"172.21.15.2:6811/2108456471","cluster_addr":"172.21.15.2:6813/2108456471","heartbeat_back_addr":"172.21.15.2:6817/2108456471","heartbeat_front_addr":"172.21.15.2:6815/2108456471","state":["exists","up"]},{"osd":2,"uuid":"035fa366-8803-4c89-a15c-3b94d2275732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6819","nonce":1651599325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6821","nonce":1651599325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6825","nonce":1651599325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1651599325},{"type":"v1","addr":"172.21.15.2:6823","nonce":1651599325}]},"public_addr":"172.21.15.2:6819/1651599325","cluster_addr":"172.21.15.2:6821/1651599325","heartbeat_back_addr":"172.21.15.2:6825/1651599325","heartbeat_front_addr":"172.21.15.2:6823/1651599325","state":["exists","up"]},{"osd":3,"uuid":"72826f52-ca60-4249-8519-a0af528d0cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6827","nonce":55264577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6829","nonce":55264577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6833","nonce":55264577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":55264577},{"type":"v1","addr":"172.21.15.2:6831","nonce":55264577}]},"public_addr":"172.21.15.2:6827/55264577","cluster_addr":"172.21.15.2:6829/55264577","heartbeat_back_addr":"172.21.15.2:6833/55264577","heartbeat_front_addr":"172.21.15.2:6831/55264577","state":["exists","up"]},{"osd":4,"uuid":"98ee6493-300c-4c52-8d2a-02f4ecd4f9bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6801","nonce":2220714878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6803","nonce":2220714878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6807","nonce":2220714878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2220714878},{"type":"v1","addr":"172.21.15.98:6805","nonce":2220714878}]},"public_addr":"172.21.15.98:6801/2220714878","cluster_addr":"172.21.15.98:6803/2220714878","heartbeat_back_addr":"172.21.15.98:6807/2220714878","heartbeat_front_addr":"172.21.15.98:6805/2220714878","state":["exists","up"]},{"osd":5,"uuid":"9f9c6145-bc4d-4870-a11c-95ccb18eb8e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6809","nonce":2911586659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6811","nonce":2911586659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6815","nonce":2911586659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2911586659},{"type":"v1","addr":"172.21.15.98:6813","nonce":2911586659}]},"public_addr":"172.21.15.98:6809/2911586659","cluster_addr":"172.21.15.98:6811/2911586659","heartbeat_back_addr":"172.21.15.98:6815/2911586659","heartbeat_front_addr":"172.21.15.98:6813/2911586659","state":["exists","up"]},{"osd":6,"uuid":"4aec8aee-aec6-4823-9920-d27608a3519e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6817","nonce":1973988038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1973988038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1973988038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1973988038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1973988038}]},"public_addr":"172.21.15.98:6817/1973988038","cluster_addr":"172.21.15.98:6819/1973988038","heartbeat_back_addr":"172.21.15.98:6823/1973988038","heartbeat_front_addr":"172.21.15.98:6821/1973988038","state":["exists","up"]},{"osd":7,"uuid":"8b27efae-537e-4f57-a0d7-bc914da043b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6825","nonce":3790380699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6827","nonce":3790380699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6831","nonce":3790380699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3790380699},{"type":"v1","addr":"172.21.15.98:6829","nonce":3790380699}]},"public_addr":"172.21.15.98:6825/3790380699","cluster_addr":"172.21.15.98:6827/3790380699","heartbeat_back_addr":"172.21.15.98:6831/3790380699","heartbeat_front_addr":"172.21.15.98:6829/3790380699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:04.636954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:28.880952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:04:53.268030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:17.422334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:05:38.397942+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:00.971584+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:23.674182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T15:06:47.038045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1096752096":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/1844860704":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/604203265":"2024-02-14T15:07:07.774359+0000","172.21.15.2:0/291642484":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/1730337629":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/219490389":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6801/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:6800/1111087951":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6800/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/3958364651":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6801/1495240999":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/2721686623":"2024-02-14T15:02:22.695373+0000","172.21.15.2:6801/1111087951":"2024-02-14T15:07:07.774359+0000","172.21.15.2:6800/1741139466":"2024-02-14T15:02:45.104290+0000","172.21.15.2:0/2705129878":"2024-02-14T15:02:22.695373+0000","172.21.15.2:0/760776131":"2024-02-14T15:02:22.695373+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T15:07:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:33 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:33 smithi098 ceph-mon[25694]: Deploying daemon node-exporter.b on smithi098 2024-02-13T15:07:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:33 smithi098 ceph-mon[25694]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:33.332 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:33 smithi098 bash[41066]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-13T15:07:33.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:33.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[24754]: Deploying daemon node-exporter.b on smithi098 2024-02-13T15:07:33.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[24754]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[29232]: Deploying daemon node-exporter.b on smithi098 2024-02-13T15:07:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:33 smithi002 ceph-mon[29232]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:33.832 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:33 smithi098 bash[41066]: Getting image source signatures 2024-02-13T15:07:33.832 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:33 smithi098 bash[41066]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-13T15:07:33.832 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:33 smithi098 bash[41066]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-13T15:07:33.832 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:33 smithi098 bash[41066]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-13T15:07:34.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-13T15:07:34.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-13T15:07:34.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-13T15:07:34.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-13T15:07:34.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-13T15:07:34.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-13T15:07:34.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-13T15:07:34.065 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-13T15:07:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:34 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2743486561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:34.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:34 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2743486561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:34 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2743486561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T15:07:34.440 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.192 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.196 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.197 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.198 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.204 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.217 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:35.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:35 smithi098 ceph-mon[25694]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:35.583 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:35 smithi098 bash[41066]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-13T15:07:35.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:35 smithi002 ceph-mon[24754]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:35.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:35 smithi002 ceph-mon[29232]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:35.950 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:35 smithi098 bash[41066]: Writing manifest to image destination 2024-02-13T15:07:35.950 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:35 smithi098 podman[41066]: 2024-02-13 15:07:35.884164324 +0000 UTC m=+2.920296973 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-13T15:07:35.950 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:35 smithi098 podman[41066]: 2024-02-13 15:07:35.949311691 +0000 UTC m=+2.985444341 container create 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:07:36.203 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 podman[41066]: 2024-02-13 15:07:36.1264979 +0000 UTC m=+3.162630544 container init 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:07:36.203 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 podman[41066]: 2024-02-13 15:07:36.129352812 +0000 UTC m=+3.165485455 container start 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:07:36.203 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.133Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.133Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.133Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.133Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T15:07:36.204 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T15:07:36.205 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T15:07:36.206 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T15:07:36.207 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[41117]: ts=2024-02-13T15:07:36.134Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T15:07:36.208 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 bash[41066]: 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 2024-02-13T15:07:36.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:07:36 smithi098 systemd[1]: Started Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:07:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:37 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:37 smithi098 ceph-mon[25694]: Deploying daemon alertmanager.a on smithi002 2024-02-13T15:07:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:37 smithi098 ceph-mon[25694]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[24754]: Deploying daemon alertmanager.a on smithi002 2024-02-13T15:07:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[24754]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:37.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:37.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[29232]: Deploying daemon alertmanager.a on smithi002 2024-02-13T15:07:37.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:37 smithi002 ceph-mon[29232]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:39.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:39 smithi098 ceph-mon[25694]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:39 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:40.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:39 smithi002 ceph-mon[24754]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:40.007 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:39 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:40.007 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:39 smithi002 ceph-mon[29232]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:40.007 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:39 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:07:40.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:07:41.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:41 smithi002 ceph-mon[24754]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:41.708 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:41 smithi002 ceph-mon[29232]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:41 smithi098 ceph-mon[25694]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:42.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:42] "GET /metrics HTTP/1.1" 200 191462 "" "Prometheus/2.33.4" 2024-02-13T15:07:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:43 smithi098 ceph-mon[25694]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:44.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:43 smithi002 ceph-mon[24754]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:44.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:43 smithi002 ceph-mon[29232]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:45.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:45 smithi002 ceph-mon[24754]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:45.553 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:45 smithi002 ceph-mon[29232]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:45.578 INFO:teuthology.orchestra.run.smithi002.stdout:128849018905 2024-02-13T15:07:45.578 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-13T15:07:45.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:45 smithi098 ceph-mon[25694]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:47.690 INFO:teuthology.orchestra.run.smithi002.stdout:34359738413 2024-02-13T15:07:47.690 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-13T15:07:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:47 smithi098 ceph-mon[25694]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:48.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:47 smithi002 ceph-mon[24754]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:48.094 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:47 smithi002 ceph-mon[29232]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:48.693 INFO:teuthology.orchestra.run.smithi002.stdout:107374182431 2024-02-13T15:07:48.694 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-13T15:07:49.315 INFO:teuthology.orchestra.run.smithi002.stdout:197568495634 2024-02-13T15:07:49.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-13T15:07:49.841 INFO:teuthology.orchestra.run.smithi002.stdout:231928233997 2024-02-13T15:07:49.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-13T15:07:49.966 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:49 smithi002 ceph-mon[24754]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:49.966 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:49 smithi002 ceph-mon[29232]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:49 smithi098 ceph-mon[25694]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:50.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:07:52.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:51 smithi098 ceph-mon[25694]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:52.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:51 smithi002 ceph-mon[24754]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:51 smithi002 ceph-mon[29232]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:52.857 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:07:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:52] "GET /metrics HTTP/1.1" 200 191462 "" "Prometheus/2.33.4" 2024-02-13T15:07:53.081 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:53.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:53 smithi002 ceph-mon[24754]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:53.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:53 smithi002 ceph-mon[29232]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:53 smithi098 ceph-mon[25694]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:54.515 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:54.516 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:54.516 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:54.517 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:07:55.058 INFO:teuthology.orchestra.run.smithi002.stdout:55834574890 2024-02-13T15:07:55.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-13T15:07:55.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:55 smithi002 ceph-mon[24754]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:55.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:55 smithi002 ceph-mon[29232]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:163208757271 2024-02-13T15:07:55.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-13T15:07:55.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:55 smithi098 ceph-mon[25694]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:57.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:57 smithi002 ceph-mon[24754]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:57.999 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:57 smithi002 ceph-mon[29232]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:57 smithi098 ceph-mon[25694]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:07:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:07:59 smithi098 ceph-mon[25694]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:00.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:07:59 smithi002 ceph-mon[24754]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:00.054 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:07:59 smithi002 ceph-mon[29232]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:00.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:07:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:07:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:01.101 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:01.102 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:01.129 INFO:teuthology.orchestra.run.smithi002.stdout:77309411367 2024-02-13T15:08:01.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-13T15:08:01.551 INFO:teuthology.orchestra.run.smithi002.stdout:107374182434 2024-02-13T15:08:01.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:01 smithi002 ceph-mon[24754]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:01.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:01 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2723149163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T15:08:01.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:01 smithi002 ceph-mon[29232]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:01.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:01 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2723149163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T15:08:02.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:01 smithi098 ceph-mon[25694]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:02.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:01 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2723149163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T15:08:02.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182431 got 107374182434 for osd.3 2024-02-13T15:08:02.307 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:02.831 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:02] "GET /metrics HTTP/1.1" 200 191456 "" "Prometheus/2.33.4" 2024-02-13T15:08:03.872 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:03 smithi002 ceph-mon[24754]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:03.873 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:03 smithi002 ceph-mon[29232]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:03 smithi098 ceph-mon[25694]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:04.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:04 smithi002 podman[46516]: 2024-02-13 15:08:04.089301473 +0000 UTC m=+1.456720332 volume create e375b41c6850e191e92d0f075126a3c8a7356d77657425bacc0af9c0128300a4 2024-02-13T15:08:04.173 INFO:teuthology.orchestra.run.smithi002.stdout:34359738417 2024-02-13T15:08:04.450 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:04 smithi002 podman[46516]: 2024-02-13 15:08:04.240045435 +0000 UTC m=+1.607464334 container create b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:04.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:04 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/211610232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T15:08:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:04 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/211610232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T15:08:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:04 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/211610232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T15:08:05.993 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-mon[24754]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:05.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-mon[29232]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:05.993 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 podman[46516]: 2024-02-13 15:08:05.826800935 +0000 UTC m=+3.194219796 container init b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:05.993 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 podman[46516]: 2024-02-13 15:08:05.830130112 +0000 UTC m=+3.197548962 container start b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:05.993 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.842Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-13T15:08:05.993 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.842Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.843Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.845Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.872Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.872Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.875Z caller=main.go:518 msg=Listening address=:9093 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:05.875Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 bash[46516]: b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 2024-02-13T15:08:05.994 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:05 smithi002 systemd[1]: Started Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:08:06.047 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:05 smithi098 ceph-mon[25694]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:06.646 INFO:teuthology.orchestra.run.smithi002.stdout:163208757273 2024-02-13T15:08:07.323 INFO:teuthology.orchestra.run.smithi002.stdout:197568495638 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3605751446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T15:08:07.323 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:07.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:08:07.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:07.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:07.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:07.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3605751446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:07 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3605751446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T15:08:07.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:07.846Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000605729s 2024-02-13T15:08:07.946 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738417 for osd.0 2024-02-13T15:08:07.947 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:08.191 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:08 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/842824176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T15:08:08.191 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:08 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/842824176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T15:08:08.312 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:08 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/842824176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T15:08:08.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757273 for osd.5 2024-02-13T15:08:08.824 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:09.214 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:09.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[24754]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:09.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[29232]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:08:09.830 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:08:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-mon[25694]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:08:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:08:10.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495638 for osd.6 2024-02-13T15:08:10.165 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:10.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:10.808 INFO:teuthology.orchestra.run.smithi002.stdout:231928234002 2024-02-13T15:08:11.429 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928234002 for osd.7 2024-02-13T15:08:11.429 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:11.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:11 smithi002 ceph-mon[24754]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:11.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:11 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2862405094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T15:08:11.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:11 smithi002 ceph-mon[29232]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:11.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:11 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2862405094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T15:08:12.070 INFO:teuthology.orchestra.run.smithi002.stdout:128849018912 2024-02-13T15:08:12.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:11 smithi098 ceph-mon[25694]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:12.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:11 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2862405094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T15:08:12.668 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:12] "GET /metrics HTTP/1.1" 200 191459 "" "Prometheus/2.33.4" 2024-02-13T15:08:12.668 INFO:teuthology.orchestra.run.smithi002.stdout:55834574894 2024-02-13T15:08:12.921 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:12 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3503513441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T15:08:12.921 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:12 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/424557086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T15:08:12.921 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:12 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3503513441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T15:08:12.921 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:12 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/424557086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T15:08:12.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018912 for osd.4 2024-02-13T15:08:12.922 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:12 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3503513441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T15:08:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:12 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/424557086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T15:08:13.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:13 smithi002 ceph-mon[24754]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:13.965 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:13 smithi002 ceph-mon[29232]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:13 smithi098 ceph-mon[25694]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:14.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574894 for osd.1 2024-02-13T15:08:14.384 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:15.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:15 smithi002 ceph-mon[24754]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:15.551 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:15 smithi002 ceph-mon[29232]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:15.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:15 smithi098 ceph-mon[25694]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:15.856 INFO:teuthology.orchestra.run.smithi002.stdout:77309411370 2024-02-13T15:08:16.126 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:15.848Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003152226s 2024-02-13T15:08:16.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:16 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1302331803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T15:08:16.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:16 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1302331803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T15:08:16.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:16 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1302331803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T15:08:17.119 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411370 for osd.2 2024-02-13T15:08:17.119 DEBUG:teuthology.parallel:result is None 2024-02-13T15:08:17.119 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-13T15:08:17.120 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T15:08:17.346 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:17 smithi002 ceph-mon[24754]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:17.346 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:17 smithi002 ceph-mon[29232]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:17.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:17 smithi098 ceph-mon[25694]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:17.884 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:18.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:18.930 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:19.649 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:08:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-02-13T15:08:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:19 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:19 smithi098 ceph-mon[25694]: Deploying daemon grafana.a on smithi098 2024-02-13T15:08:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:19 smithi098 ceph-mon[25694]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:20.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:19 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:20.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:20.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[24754]: Deploying daemon grafana.a on smithi098 2024-02-13T15:08:20.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[24754]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:20.119 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:08:20.120 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[29232]: Deploying daemon grafana.a on smithi098 2024-02-13T15:08:20.120 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:19 smithi002 ceph-mon[29232]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:20.412 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-02-13T15:08:18.643202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48892,"kb_used_data":6076,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912324,"statfs":{"total":767960285184,"available":767910219776,"internally_reserved":0,"allocated":6221824,"data_stored":3487541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002025"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-13T15:07:08.722776+0000","last_change":"2024-02-13T15:06:52.983478+0000","last_active":"2024-02-13T15:07:08.722776+0000","last_peered":"2024-02-13T15:07:08.722776+0000","last_clean":"2024-02-13T15:07:08.722776+0000","last_became_active":"2024-02-13T15:06:52.983078+0000","last_became_peered":"2024-02-13T15:06:52.983078+0000","last_unstale":"2024-02-13T15:07:08.722776+0000","last_undegraded":"2024-02-13T15:07:08.722776+0000","last_fullsized":"2024-02-13T15:07:08.722776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T15:04:55.988936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T15:04:55.988936+0000","last_clean_scrub_stamp":"2024-02-13T15:04:55.988936+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:05:07.111922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.57899999999999996}]},{"osd":1,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63400000000000001}]},{"osd":2,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55700000000000005}]},{"osd":4,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48299999999999998}]},{"osd":5,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.77700000000000002}]},{"osd":6,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.624}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.192,"5min":11.192,"15min":11.192},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.57499999999999996}]},{"osd":3,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.59899999999999998}]},{"osd":4,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.62},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.60599999999999998}]},{"osd":5,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.72499999999999998}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.747}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.32000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48399999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.33400000000000002},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.56699999999999995}]},{"osd":3,"last update":"Tue Feb 13 15:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47099999999999997,"15min":0.46899999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Feb 13 15:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83999999999999997,"5min":0.87,"15min":0.87},"last":0.66600000000000004}]},{"osd":5,"last update":"Tue Feb 13 15:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71999999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.627}]},{"osd":6,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73799999999999999}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.75}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 15:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.34699999999999998,"15min":0.32700000000000001},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53300000000000003,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.625,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.40200000000000002}]},{"osd":2,"last update":"Tue Feb 13 15:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.54200000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.33800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Feb 13 15:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56399999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.443}]},{"osd":4,"last update":"Tue Feb 13 15:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82199999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.754}]},{"osd":5,"last update":"Tue Feb 13 15:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.44500000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Feb 13 15:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Feb 13 15:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.77100000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.37},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.40600000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.30599999999999999}]},{"osd":1,"last update":"Tue Feb 13 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.41399999999999998,"15min":0.39500000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.29799999999999999}]},{"osd":3,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue Feb 13 15:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.47199999999999998}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Tue Feb 13 15:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45600000000000002}]},{"osd":7,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.498},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57999999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.35799999999999998}]},{"osd":1,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.35699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.373}]},{"osd":2,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.625,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46899999999999997,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.51300000000000001}]},{"osd":4,"last update":"Tue Feb 13 15:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":1.8779999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73899999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52500000000000002}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.68500000000000005}]},{"osd":6,"last update":"Tue Feb 13 15:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45100000000000001}]},{"osd":7,"last update":"Tue Feb 13 15:07:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":1.867}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63400000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.67400000000000004,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59599999999999997}]},{"osd":3,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.629,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55000000000000004}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42199999999999999},"min":{"1min":0.318,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.436,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.376}]},{"osd":6,"last update":"Tue Feb 13 15:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.49299999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":236988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.624},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70699999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.47499999999999998}]},{"osd":2,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.40400000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73099999999999998,"5min":0.755,"15min":0.755},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69499999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.44900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.497}]},{"osd":4,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.373,"15min":0.36499999999999999},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.56100000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38100000000000001},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.442}]},{"osd":6,"last update":"Tue Feb 13 15:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53300000000000003}]},{"osd":7,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T15:08:20.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T15:08:20.634 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:21.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:20 smithi098 ceph-mon[25694]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:21.089 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:20 smithi002 ceph-mon[24754]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:21.090 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:20 smithi002 ceph-mon[29232]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:22.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:21 smithi002 ceph-mon[24754]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:22.063 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:21 smithi002 ceph-mon[29232]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:22.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:21 smithi098 ceph-mon[25694]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:22.450 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:08:22.452 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-02-13T15:08:22.749 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:22] "GET /metrics HTTP/1.1" 200 191459 "" "Prometheus/2.33.4" 2024-02-13T15:08:23.039 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-13T15:08:20.643598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48892,"kb_used_data":6076,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912324,"statfs":{"total":767960285184,"available":767910219776,"internally_reserved":0,"allocated":6221824,"data_stored":3487541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002107"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-13T15:07:08.722776+0000","last_change":"2024-02-13T15:06:52.983478+0000","last_active":"2024-02-13T15:07:08.722776+0000","last_peered":"2024-02-13T15:07:08.722776+0000","last_clean":"2024-02-13T15:07:08.722776+0000","last_became_active":"2024-02-13T15:06:52.983078+0000","last_became_peered":"2024-02-13T15:06:52.983078+0000","last_unstale":"2024-02-13T15:07:08.722776+0000","last_undegraded":"2024-02-13T15:07:08.722776+0000","last_fullsized":"2024-02-13T15:07:08.722776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T15:04:55.988936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T15:04:55.988936+0000","last_clean_scrub_stamp":"2024-02-13T15:04:55.988936+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:05:07.111922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.253}]},{"osd":1,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.33100000000000002}]},{"osd":3,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.498},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47499999999999998}]},{"osd":5,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51200000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.192,"5min":11.192,"15min":11.192},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.73399999999999999}]},{"osd":4,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.81000000000000005}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.32000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48399999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.33400000000000002},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.36799999999999999}]},{"osd":2,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.29399999999999998}]},{"osd":3,"last update":"Tue Feb 13 15:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47099999999999997,"15min":0.46899999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Tue Feb 13 15:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83999999999999997,"5min":0.87,"15min":0.87},"last":0.49199999999999999}]},{"osd":5,"last update":"Tue Feb 13 15:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71999999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.502},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.42499999999999999}]},{"osd":6,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 15:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.34699999999999998,"15min":0.32700000000000001},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53300000000000003,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.625,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Feb 13 15:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.54200000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.33800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Tue Feb 13 15:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.63},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56399999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.40100000000000002}]},{"osd":4,"last update":"Tue Feb 13 15:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82199999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52000000000000002}]},{"osd":5,"last update":"Tue Feb 13 15:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.44500000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51000000000000001}]},{"osd":6,"last update":"Tue Feb 13 15:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47199999999999998}]},{"osd":7,"last update":"Tue Feb 13 15:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59799999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.37},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.371},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.40600000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.35799999999999998}]},{"osd":1,"last update":"Tue Feb 13 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.41399999999999998,"15min":0.39500000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Feb 13 15:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Feb 13 15:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63},{"interface":"front","average":{"1min":0.623,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76000000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Feb 13 15:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71199999999999997}]},{"osd":7,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60999999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.35699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Feb 13 15:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.625,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46899999999999997,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.26200000000000001}]},{"osd":4,"last update":"Tue Feb 13 15:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73899999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53800000000000003}]},{"osd":6,"last update":"Tue Feb 13 15:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38200000000000001}]},{"osd":7,"last update":"Tue Feb 13 15:07:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71899999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.44},{"interface":"front","average":{"1min":0.495,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53900000000000003}]},{"osd":1,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63400000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69399999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.67400000000000004,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52400000000000002}]},{"osd":3,"last update":"Tue Feb 13 15:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.629,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.37,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45700000000000002}]},{"osd":5,"last update":"Tue Feb 13 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42199999999999999},"min":{"1min":0.318,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.436,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Feb 13 15:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.40000000000000002}]},{"osd":7,"last update":"Tue Feb 13 15:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56999999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":236988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46000000000000002}]},{"osd":1,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70699999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.51900000000000002}]},{"osd":2,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.499},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.40400000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73099999999999998,"5min":0.755,"15min":0.755},"last":0.48499999999999999}]},{"osd":3,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69499999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.44900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.441}]},{"osd":4,"last update":"Tue Feb 13 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.373,"15min":0.36499999999999999},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.56100000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38800000000000001,"15min":0.38100000000000001},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue Feb 13 15:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Feb 13 15:07:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T15:08:23.041 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-13T15:08:23.041 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-13T15:08:23.041 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-13T15:08:23.041 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-13T15:08:23.258 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T15:08:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:23 smithi098 ceph-mon[25694]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:23 smithi098 ceph-mon[25694]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:23 smithi002 ceph-mon[24754]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:23 smithi002 ceph-mon[24754]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:23 smithi002 ceph-mon[29232]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T15:08:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:23 smithi002 ceph-mon[29232]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:24.990 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-13T15:08:24.990 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-13T15:08:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:25 smithi002 ceph-mon[24754]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:25.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:25 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3489557758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T15:08:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:25 smithi002 ceph-mon[29232]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:25 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3489557758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T15:08:25.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:25 smithi098 ceph-mon[25694]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:25.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:25 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3489557758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T15:08:25.607 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-13T15:08:25.607 INFO:tasks.cephadm:Setup complete, yielding 2024-02-13T15:08:25.607 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-13T15:08:25.616 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T15:08:25.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-02-13T15:08:27.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:27 smithi002 ceph-mon[29232]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:27.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:27 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' 2024-02-13T15:08:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:27 smithi002 ceph-mon[24754]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:27 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' 2024-02-13T15:08:27.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:27 smithi098 ceph-mon[25694]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:27.988 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:27 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' 2024-02-13T15:08:28.277 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-13T15:08:28.286 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T15:08:28.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-13T15:08:29.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:29 smithi002 ceph-mon[24754]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:29.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:29 smithi002 ceph-mon[29232]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:29.964 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:29 smithi098 ceph-mon[25694]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:30.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:30.926 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-13T15:08:31.177 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-13T15:08:31.178 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-13T15:08:31.179 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-13T15:08:31.180 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-13T15:08:31.181 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:30 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:30+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-13T15:08:31.182 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-13T15:08:31.183 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-13T15:08:31.184 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-13T15:08:31.185 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-13T15:08:31.186 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-13T15:08:31.187 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-13T15:08:31.188 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-13T15:08:31.189 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-13T15:08:31.190 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-13T15:08:31.191 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-13T15:08:31.192 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-13T15:08:31.193 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-13T15:08:31.194 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-13T15:08:31.560 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-13T15:08:31.561 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-13T15:08:31.562 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-13T15:08:31.563 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-13T15:08:31.564 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-13T15:08:31.564 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-13T15:08:31.564 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-13T15:08:31.565 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-13T15:08:31.566 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-13T15:08:31.567 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=450.270536ms 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-13T15:08:31.568 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-13T15:08:31.569 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:08:31+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "59e2853c-0fb2-4550-af5f-c02485b4beac", 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "r", 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "current_period": "a539bfea-de88-4c68-b2d1-f94e83c35dc3", 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1 2024-02-13T15:08:31.747 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:08:31.766 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:31.766 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: osdmap e60: 8 total, 8 up, 8 in 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1034153246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: osdmap e60: 8 total, 8 up, 8 in 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1034153246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.767 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.768 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:31.768 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:31.768 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:31 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: osdmap e60: 8 total, 8 up, 8 in 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1034153246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:31.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:31.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:31 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:32.681 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-13T15:08:32.805 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:32 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T15:08:32.806 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:32 smithi098 ceph-mon[25694]: osdmap e61: 8 total, 8 up, 8 in 2024-02-13T15:08:32.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:32 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T15:08:32.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:32 smithi002 ceph-mon[24754]: osdmap e61: 8 total, 8 up, 8 in 2024-02-13T15:08:32.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:32] "GET /metrics HTTP/1.1" 200 191466 "" "Prometheus/2.33.4" 2024-02-13T15:08:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:32 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T15:08:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:32 smithi002 ceph-mon[29232]: osdmap e61: 8 total, 8 up, 8 in 2024-02-13T15:08:34.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:33 smithi098 ceph-mon[25694]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:34.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:33 smithi098 ceph-mon[25694]: osdmap e62: 8 total, 8 up, 8 in 2024-02-13T15:08:34.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:33 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:34.052 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[24754]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:34.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[24754]: osdmap e62: 8 total, 8 up, 8 in 2024-02-13T15:08:34.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:34.053 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[29232]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:08:34.053 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[29232]: osdmap e62: 8 total, 8 up, 8 in 2024-02-13T15:08:34.053 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:33 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "2bbae6a4-d036-4650-8cef-5ca92ffa27e8", 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default", 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout: "api_name": "default", 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout: "is_master": "true", 2024-02-13T15:08:34.607 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames": [], 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames_s3website": [], 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "", 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "zones": [], 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_targets": [], 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "default_placement": "", 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "59e2853c-0fb2-4550-af5f-c02485b4beac", 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_policy": { 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: "groups": [] 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:34.608 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:08:35.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:35 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:35.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:35 smithi002 ceph-mon[24754]: pgmap v49: 33 pgs: 25 active+clean, 8 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T15:08:35.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:35 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:35.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:35 smithi002 ceph-mon[29232]: pgmap v49: 33 pgs: 25 active+clean, 8 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T15:08:35.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:35 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:35.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:35 smithi098 ceph-mon[25694]: pgmap v49: 33 pgs: 25 active+clean, 8 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T15:08:35.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-13T15:08:37.954 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[24754]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-02-13T15:08:37.954 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:37.954 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:37.954 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[29232]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-02-13T15:08:37.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:37.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:37 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:37 smithi098 ceph-mon[25694]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-02-13T15:08:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:37 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:37 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:38.345 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:08:38.345 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "4a23beb7-5c38-4bc6-9781-4db35e4f1861", 2024-02-13T15:08:38.345 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2024-02-13T15:08:38.345 INFO:teuthology.orchestra.run.smithi002.stdout: "domain_root": "z.rgw.meta:root", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "control_pool": "z.rgw.control", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "log_pool": "z.rgw.log", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-13T15:08:38.346 INFO:teuthology.orchestra.run.smithi002.stdout: "otp_pool": "z.rgw.otp", 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "system_key": { 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "access_key": "", 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "secret_key": "" 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_pools": [ 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "default-placement", 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "val": { 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_classes": { 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "STANDARD": { 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:38.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: "index_type": 0 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "59e2853c-0fb2-4550-af5f-c02485b4beac", 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout: "notif_pool": "z.rgw.log:notif" 2024-02-13T15:08:38.348 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:08:38.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:38 smithi002 ceph-mon[24754]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:08:38.864 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:38 smithi002 ceph-mon[24754]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:08:38.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:38 smithi002 ceph-mon[29232]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:08:38.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:38 smithi002 ceph-mon[29232]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:08:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:38 smithi098 ceph-mon[25694]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:08:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:38 smithi098 ceph-mon[25694]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:08:39.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-13T15:08:39.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:39 smithi098 ceph-mon[25694]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-13T15:08:40.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:39 smithi002 ceph-mon[24754]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-13T15:08:40.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:39 smithi002 ceph-mon[29232]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-13T15:08:40.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:41.710 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:41 smithi002 systemd[1]: Stopping Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:08:41.710 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:41 smithi002 bash[49127]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T15:08:42.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:41 smithi002 ceph-mon[24754]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T15:08:42.055 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:41 smithi002 ceph-mon[29232]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T15:08:42.071 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:41 smithi098 ceph-mon[25694]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 802 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T15:08:42.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:42] "GET /metrics HTTP/1.1" 200 192994 "" "Prometheus/2.33.4" 2024-02-13T15:08:42.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[46603]: level=info ts=2024-02-13T15:08:42.357Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:08:42.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:42 smithi002 podman[49154]: 2024-02-13 15:08:42.378219334 +0000 UTC m=+0.745151706 container died b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:43.018 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:42 smithi002 podman[49154]: 2024-02-13 15:08:42.815858944 +0000 UTC m=+1.182791317 container cleanup b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:43.018 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:42 smithi002 bash[49154]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T15:08:43.377 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 podman[49186]: 2024-02-13 15:08:43.016920642 +0000 UTC m=+0.633717763 container remove b5e67f680aa8ff521eb2cd190e7023aecf061332da6f6978cbfc94f93bc94c29 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:43.377 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 podman[49186]: 2024-02-13 15:08:43.092617757 +0000 UTC m=+0.709414873 volume remove e375b41c6850e191e92d0f075126a3c8a7356d77657425bacc0af9c0128300a4 2024-02-13T15:08:43.872 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:43 smithi002 ceph-mon[24754]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-13T15:08:43.872 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:43 smithi002 ceph-mon[29232]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-13T15:08:43.873 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 bash[49198]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T15:08:43.873 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-13T15:08:43.873 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 systemd[1]: Stopped Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:08:43.873 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:43 smithi002 systemd[1]: Starting Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:08:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:43 smithi098 ceph-mon[25694]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-02-13T15:08:44.548 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 podman[49342]: 2024-02-13 15:08:44.241286399 +0000 UTC m=+0.107828643 volume create a881718371aa375f46d32d52cacdd99e07e615061d18f7a9f4c5947d62fa763c 2024-02-13T15:08:44.548 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 podman[49342]: 2024-02-13 15:08:44.151230249 +0000 UTC m=+0.017772507 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-13T15:08:44.548 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 podman[49342]: 2024-02-13 15:08:44.341696032 +0000 UTC m=+0.208238321 container create 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 podman[49342]: 2024-02-13 15:08:44.652588103 +0000 UTC m=+0.519130359 container init 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 podman[49342]: 2024-02-13 15:08:44.655267782 +0000 UTC m=+0.521810027 container start 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.665Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.665Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.666Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.668Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.693Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.693Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.695Z caller=main.go:518 msg=Listening address=:9093 2024-02-13T15:08:45.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:44.695Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-13T15:08:45.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 bash[49342]: 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c 2024-02-13T15:08:45.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:44 smithi002 systemd[1]: Started Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: osdmap e63: 8 total, 8 up, 8 in 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:46.029 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[24754]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: osdmap e63: 8 total, 8 up, 8 in 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:08:46.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:45 smithi002 ceph-mon[29232]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: osdmap e63: 8 total, 8 up, 8 in 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:08:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:45 smithi098 ceph-mon[25694]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:08:46.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:46.668Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000170682s 2024-02-13T15:08:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:46 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-13T15:08:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:46 smithi098 ceph-mon[25694]: osdmap e64: 8 total, 8 up, 8 in 2024-02-13T15:08:47.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:46 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-13T15:08:47.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:46 smithi002 ceph-mon[24754]: osdmap e64: 8 total, 8 up, 8 in 2024-02-13T15:08:47.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:46 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-13T15:08:47.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:46 smithi002 ceph-mon[29232]: osdmap e64: 8 total, 8 up, 8 in 2024-02-13T15:08:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:47 smithi098 ceph-mon[25694]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 639 B/s wr, 4 op/s 2024-02-13T15:08:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:47 smithi098 ceph-mon[25694]: osdmap e65: 8 total, 8 up, 8 in 2024-02-13T15:08:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:47 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:48.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:47 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[24754]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 639 B/s wr, 4 op/s 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[24754]: osdmap e65: 8 total, 8 up, 8 in 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[29232]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 639 B/s wr, 4 op/s 2024-02-13T15:08:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[29232]: osdmap e65: 8 total, 8 up, 8 in 2024-02-13T15:08:48.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1720496898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:48.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:47 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-13T15:08:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:48 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-13T15:08:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:48 smithi098 ceph-mon[25694]: osdmap e66: 8 total, 8 up, 8 in 2024-02-13T15:08:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:48 smithi098 ceph-mon[25694]: osdmap e67: 8 total, 8 up, 8 in 2024-02-13T15:08:49.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:48 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:49.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:48 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[24754]: osdmap e66: 8 total, 8 up, 8 in 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[24754]: osdmap e67: 8 total, 8 up, 8 in 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[29232]: osdmap e66: 8 total, 8 up, 8 in 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[29232]: osdmap e67: 8 total, 8 up, 8 in 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:48 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T15:08:50.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:08:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-mon[25694]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:08:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-13T15:08:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-mon[25694]: osdmap e68: 8 total, 8 up, 8 in 2024-02-13T15:08:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:50.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:49 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[24754]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[24754]: osdmap e68: 8 total, 8 up, 8 in 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[29232]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:08:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-13T15:08:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[29232]: osdmap e68: 8 total, 8 up, 8 in 2024-02-13T15:08:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:49 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "2d300a60-b9e1-4830-bd2f-d91872af41d2", 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "predecessor_uuid": "a539bfea-de88-4c68-b2d1-f94e83c35dc3", 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_status": [], 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "period_map": { 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "2d300a60-b9e1-4830-bd2f-d91872af41d2", 2024-02-13T15:08:51.806 INFO:teuthology.orchestra.run.smithi002.stdout: "zonegroups": [ 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "2bbae6a4-d036-4650-8cef-5ca92ffa27e8", 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default", 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "api_name": "default", 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "is_master": "true", 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames": [], 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames_s3website": [], 2024-02-13T15:08:51.807 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "4a23beb7-5c38-4bc6-9781-4db35e4f1861", 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "zones": [ 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "4a23beb7-5c38-4bc6-9781-4db35e4f1861", 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "log_meta": "false", 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "log_data": "false", 2024-02-13T15:08:51.808 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_index_max_shards": 11, 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "read_only": "false", 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "tier_type": "", 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_from_all": "true", 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_from": [], 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_zone": "" 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:08:51.809 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_targets": [ 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default-placement", 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_classes": [ 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "STANDARD" 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "default_placement": "default-placement", 2024-02-13T15:08:51.810 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "59e2853c-0fb2-4550-af5f-c02485b4beac", 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_policy": { 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: "groups": [] 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: "short_zone_ids": [ 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "4a23beb7-5c38-4bc6-9781-4db35e4f1861", 2024-02-13T15:08:51.811 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 3978376973 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zonegroup": "2bbae6a4-d036-4650-8cef-5ca92ffa27e8", 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "4a23beb7-5c38-4bc6-9781-4db35e4f1861", 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "period_config": { 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_quota": { 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false, 2024-02-13T15:08:51.812 INFO:teuthology.orchestra.run.smithi002.stdout: "check_on_raw": false, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size": -1, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size_kb": 0, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_objects": -1 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "user_quota": { 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "check_on_raw": false, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size": -1, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size_kb": 0, 2024-02-13T15:08:51.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max_objects": -1 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "user_ratelimit": { 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_ratelimit": { 2024-02-13T15:08:51.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "anonymous_ratelimit": { 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-02-13T15:08:51.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "59e2853c-0fb2-4550-af5f-c02485b4beac", 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_name": "r", 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_epoch": 2 2024-02-13T15:08:51.816 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:08:51.991 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:51 smithi098 ceph-mon[25694]: pgmap v63: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:51.991 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:51 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T15:08:51.991 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:51 smithi098 ceph-mon[25694]: osdmap e69: 8 total, 8 up, 8 in 2024-02-13T15:08:51.991 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:51 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:51.991 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:51 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:52.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[24754]: pgmap v63: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[24754]: osdmap e69: 8 total, 8 up, 8 in 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[29232]: pgmap v63: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[29232]: osdmap e69: 8 total, 8 up, 8 in 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2446558211' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:51 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-13T15:08:52.578 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-02-13T15:08:52.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:52 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-13T15:08:52.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:52 smithi002 ceph-mon[24754]: osdmap e70: 8 total, 8 up, 8 in 2024-02-13T15:08:52.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:08:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:52] "GET /metrics HTTP/1.1" 200 192994 "" "Prometheus/2.33.4" 2024-02-13T15:08:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:52 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-13T15:08:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:52 smithi002 ceph-mon[29232]: osdmap e70: 8 total, 8 up, 8 in 2024-02-13T15:08:53.039 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:52 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-13T15:08:53.039 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:52 smithi098 ceph-mon[25694]: osdmap e70: 8 total, 8 up, 8 in 2024-02-13T15:08:53.301 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 bash[42482]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:08:53.577 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:08:53.577 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-13T15:08:53.577 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-13T15:08:53.577 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-13T15:08:53.577 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.300Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.301Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.301Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[40831]: ts=2024-02-13T15:08:53.301Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-13T15:08:53.578 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 podman[42495]: 2024-02-13 15:08:53.310716963 +0000 UTC m=+0.099724293 container died 5c3fc01f3630d500792ad6241f38586824fec75be7155d4979fdcad9e61f3e8d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:53 smithi098 ceph-mon[25694]: pgmap v66: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:53.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 podman[42495]: 2024-02-13 15:08:53.674684528 +0000 UTC m=+0.463691869 container cleanup 5c3fc01f3630d500792ad6241f38586824fec75be7155d4979fdcad9e61f3e8d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:53.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 bash[42495]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T15:08:53.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 bash[42520]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:08:54.098 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:53 smithi002 ceph-mon[29232]: pgmap v66: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:54.099 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:53 smithi002 ceph-mon[24754]: pgmap v66: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 podman[42509]: 2024-02-13 15:08:53.934421857 +0000 UTC m=+0.625270291 container remove 5c3fc01f3630d500792ad6241f38586824fec75be7155d4979fdcad9e61f3e8d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 systemd[1]: Stopped Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:53 smithi098 systemd[1]: Starting Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 podman[42590]: 2024-02-13 15:08:54.067970737 +0000 UTC m=+0.020155851 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-13T15:08:54.294 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 podman[42590]: 2024-02-13 15:08:54.169081147 +0000 UTC m=+0.121266286 container create 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 podman[42590]: 2024-02-13 15:08:54.39633465 +0000 UTC m=+0.348519758 container init 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 podman[42590]: 2024-02-13 15:08:54.398819614 +0000 UTC m=+0.351004705 container start 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi098 (none))" 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-13T15:08:54.666 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.422Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.424Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.424Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.426Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.427Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.427Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.713µs 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.427Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 bash[42590]: 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 systemd[1]: Started Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.661Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.662Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-13T15:08:54.667 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.662Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.799µs wal_replay_duration=234.650929ms total_replay_duration=234.693013ms 2024-02-13T15:08:54.868 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled rgw.foo update... 2024-02-13T15:08:55.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.665Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-13T15:08:55.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.665Z caller=main.go:947 level=info msg="TSDB started" 2024-02-13T15:08:55.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.665Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-13T15:08:55.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.679Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=13.778327ms db_storage=767ns remote_storage=1.253µs web_handler=356ns query_engine=721ns scrape=379.09µs scrape_sd=22.631µs notify=19.094µs notify_sd=6.62µs rules=13.114309ms 2024-02-13T15:08:55.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:08:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:08:54.679Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-13T15:08:55.126 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:08:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:08:54.669Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00159223s 2024-02-13T15:08:55.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch apply rgw smpl' 2024-02-13T15:08:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:55.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:55.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T15:08:55.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:55 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.861 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.98:9095"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.862 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.98:3000"}]: dispatch 2024-02-13T15:08:55.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:55.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:08:55.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:08:55.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T15:08:55.863 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:55 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:56.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:56 smithi098 ceph-mon[25694]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:08:56.815 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:56 smithi098 ceph-mon[25694]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:08:56.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:56 smithi002 ceph-mon[24754]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:08:56.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:56 smithi002 ceph-mon[24754]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:08:56.840 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:56 smithi002 ceph-mon[29232]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:08:56.840 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:56 smithi002 ceph-mon[29232]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:08:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:57 smithi098 ceph-mon[25694]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-13T15:08:57.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:57 smithi002 ceph-mon[29232]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-13T15:08:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:57 smithi002 ceph-mon[24754]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-13T15:08:58.357 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled rgw.smpl update... 2024-02-13T15:08:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:58 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:58 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:58 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:58 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:58.992 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:58 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:58.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:58 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:08:59.761 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph osd pool create foo' 2024-02-13T15:08:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:59 smithi098 ceph-mon[25694]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:08:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:59 smithi098 ceph-mon[25694]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:08:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:08:59 smithi098 ceph-mon[25694]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-13T15:09:00.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[24754]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:09:00.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[24754]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:09:00.018 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[24754]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-13T15:09:00.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[29232]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:09:00.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[29232]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:09:00.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:08:59 smithi002 ceph-mon[29232]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-13T15:09:00.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:08:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:08:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[24754]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 722 B/s rd, 722 B/s wr, 1 op/s 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:09:01.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:01 smithi002 ceph-mon[29232]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 722 B/s rd, 722 B/s wr, 1 op/s 2024-02-13T15:09:01.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: Saving service rgw.foo spec with placement count:2 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:09:01.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:01 smithi098 ceph-mon[25694]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 722 B/s rd, 722 B/s wr, 1 op/s 2024-02-13T15:09:02.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:02] "GET /metrics HTTP/1.1" 200 196785 "" "Prometheus/2.33.4" 2024-02-13T15:09:02.974 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'foo' created 2024-02-13T15:09:02.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:02 smithi002 ceph-mon[29232]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 656 B/s wr, 1 op/s 2024-02-13T15:09:02.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:02 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-13T15:09:02.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:02 smithi002 ceph-mon[24754]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 656 B/s wr, 1 op/s 2024-02-13T15:09:02.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:02 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-13T15:09:03.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:02 smithi098 ceph-mon[25694]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 656 B/s wr, 1 op/s 2024-02-13T15:09:03.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:02 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-13T15:09:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:03 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-13T15:09:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:03 smithi098 ceph-mon[25694]: osdmap e71: 8 total, 8 up, 8 in 2024-02-13T15:09:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:03 smithi098 ceph-mon[25694]: osdmap e72: 8 total, 8 up, 8 in 2024-02-13T15:09:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-13T15:09:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[24754]: osdmap e71: 8 total, 8 up, 8 in 2024-02-13T15:09:04.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[24754]: osdmap e72: 8 total, 8 up, 8 in 2024-02-13T15:09:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/77275780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-13T15:09:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[29232]: osdmap e71: 8 total, 8 up, 8 in 2024-02-13T15:09:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:03 smithi002 ceph-mon[29232]: osdmap e72: 8 total, 8 up, 8 in 2024-02-13T15:09:04.411 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'rbd pool init foo' 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: osdmap e73: 8 total, 8 up, 8 in 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:05.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:05 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: osdmap e73: 8 total, 8 up, 8 in 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:05.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: osdmap e73: 8 total, 8 up, 8 in 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:05.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:05.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:05 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:06.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:06 smithi098 ceph-mon[25694]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:09:06.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:06 smithi002 ceph-mon[24754]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:09:06.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:06 smithi002 ceph-mon[29232]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:09:07.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[24754]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:07.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3250965894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:07.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:07.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[29232]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:07.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3250965894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:07.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:07 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:07 smithi098 ceph-mon[25694]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:07 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3250965894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:07 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: osdmap e74: 8 total, 8 up, 8 in 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:08.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[24754]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: osdmap e74: 8 total, 8 up, 8 in 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:08.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:08.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:08.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:08 smithi002 ceph-mon[29232]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: osdmap e74: 8 total, 8 up, 8 in 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: Saving service rgw.smpl spec with placement count:2 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:08.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:08.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:08.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:08 smithi098 ceph-mon[25694]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:09:09.620 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: osdmap e75: 8 total, 8 up, 8 in 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: osdmap e75: 8 total, 8 up, 8 in 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:09.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:09:09.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: osdmap e75: 8 total, 8 up, 8 in 2024-02-13T15:09:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-13T15:09:09.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-13T15:09:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-02-13T15:09:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:09:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:09:09.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:10.150 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch apply iscsi foo u p' 2024-02-13T15:09:10.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:10.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-13T15:09:10.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-13T15:09:10.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-13T15:09:10.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-13T15:09:10.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[24754]: osdmap e76: 8 total, 8 up, 8 in 2024-02-13T15:09:10.694 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-13T15:09:10.694 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-13T15:09:10.694 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-13T15:09:10.694 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-13T15:09:10.694 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:10 smithi002 ceph-mon[29232]: osdmap e76: 8 total, 8 up, 8 in 2024-02-13T15:09:10.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-13T15:09:10.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-13T15:09:10.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-13T15:09:10.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:10 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-02-13T15:09:10.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:10 smithi098 ceph-mon[25694]: osdmap e76: 8 total, 8 up, 8 in 2024-02-13T15:09:11.483 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: osdmap e77: 8 total, 8 up, 8 in 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: pgmap v82: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:11.484 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:11 smithi098 ceph-mon[25694]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: osdmap e77: 8 total, 8 up, 8 in 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: pgmap v82: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[24754]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: osdmap e77: 8 total, 8 up, 8 in 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: pgmap v82: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 278 KiB/s rd, 8.7 KiB/s wr, 530 op/s 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:11.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:11 smithi002 ceph-mon[29232]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:09:12.718 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled iscsi.foo update... 2024-02-13T15:09:12.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:12] "GET /metrics HTTP/1.1" 200 206242 "" "Prometheus/2.33.4" 2024-02-13T15:09:13.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'sleep 120' 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 210 KiB/s rd, 6.6 KiB/s wr, 400 op/s 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: Saving service iscsi.foo spec with placement count:1 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:13.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 210 KiB/s rd, 6.6 KiB/s wr, 400 op/s 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: Saving service iscsi.foo spec with placement count:1 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:13.847 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:13.848 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:13.848 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:13 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 210 KiB/s rd, 6.6 KiB/s wr, 400 op/s 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: Saving service iscsi.foo spec with placement count:1 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:14.013 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:13 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:14.941 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:14 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:14.942 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:14 smithi098 ceph-mon[25694]: osdmap e78: 8 total, 8 up, 8 in 2024-02-13T15:09:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:14 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:14 smithi002 ceph-mon[24754]: osdmap e78: 8 total, 8 up, 8 in 2024-02-13T15:09:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:14 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:14 smithi002 ceph-mon[29232]: osdmap e78: 8 total, 8 up, 8 in 2024-02-13T15:09:15.864 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:15 smithi002 ceph-mon[24754]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 401 KiB/s rd, 6.3 KiB/s wr, 688 op/s; 7 B/s, 0 objects/s recovering 2024-02-13T15:09:15.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:15 smithi002 ceph-mon[29232]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 401 KiB/s rd, 6.3 KiB/s wr, 688 op/s; 7 B/s, 0 objects/s recovering 2024-02-13T15:09:16.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:15 smithi098 ceph-mon[25694]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 401 KiB/s rd, 6.3 KiB/s wr, 688 op/s; 7 B/s, 0 objects/s recovering 2024-02-13T15:09:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:17 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:17 smithi002 ceph-mon[24754]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 331 KiB/s rd, 5.2 KiB/s wr, 566 op/s; 6 B/s, 0 objects/s recovering 2024-02-13T15:09:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:17 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:17 smithi002 ceph-mon[29232]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 331 KiB/s rd, 5.2 KiB/s wr, 566 op/s; 6 B/s, 0 objects/s recovering 2024-02-13T15:09:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:17 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:17 smithi098 ceph-mon[25694]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 331 KiB/s rd, 5.2 KiB/s wr, 566 op/s; 6 B/s, 0 objects/s recovering 2024-02-13T15:09:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:18.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:18.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:18.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:09:18.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T15:09:18.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:18 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:19.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:09:19.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T15:09:19.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:18 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:19 smithi098 ceph-mon[25694]: Checking pool "foo" exists for service iscsi.foo 2024-02-13T15:09:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:19 smithi098 ceph-mon[25694]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:09:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:19 smithi098 ceph-mon[25694]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 370 B/s wr, 242 op/s; 5 B/s, 0 objects/s recovering 2024-02-13T15:09:19.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:19 smithi098 ceph-mon[25694]: mgrmap e21: y(active, since 2m), standbys: x 2024-02-13T15:09:20.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[24754]: Checking pool "foo" exists for service iscsi.foo 2024-02-13T15:09:20.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[24754]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:09:20.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[24754]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 370 B/s wr, 242 op/s; 5 B/s, 0 objects/s recovering 2024-02-13T15:09:20.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[24754]: mgrmap e21: y(active, since 2m), standbys: x 2024-02-13T15:09:20.065 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[29232]: Checking pool "foo" exists for service iscsi.foo 2024-02-13T15:09:20.065 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[29232]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:09:20.065 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[29232]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 370 B/s wr, 242 op/s; 5 B/s, 0 objects/s recovering 2024-02-13T15:09:20.065 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:19 smithi002 ceph-mon[29232]: mgrmap e21: y(active, since 2m), standbys: x 2024-02-13T15:09:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:19 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:21.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[24754]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/4210411864' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[29232]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:21.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:21.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:21 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/4210411864' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:09:22.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:21 smithi098 ceph-mon[25694]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:22.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:22.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:22.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:22.062 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:21 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/4210411864' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:09:22.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:22 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]: dispatch 2024-02-13T15:09:22.819 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:22] "GET /metrics HTTP/1.1" 200 206242 "" "Prometheus/2.33.4" 2024-02-13T15:09:22.819 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:22 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]: dispatch 2024-02-13T15:09:23.072 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:22 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]: dispatch 2024-02-13T15:09:23.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:09:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:09:23.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:09:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:09:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:23 smithi098 ceph-mon[25694]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:23 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]': finished 2024-02-13T15:09:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:23 smithi098 ceph-mon[25694]: osdmap e79: 8 total, 8 up, 8 in 2024-02-13T15:09:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:23 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]: dispatch 2024-02-13T15:09:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[24754]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]': finished 2024-02-13T15:09:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[24754]: osdmap e79: 8 total, 8 up, 8 in 2024-02-13T15:09:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]: dispatch 2024-02-13T15:09:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[29232]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 511 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-02-13T15:09:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1497254650' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1096752096"}]': finished 2024-02-13T15:09:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[29232]: osdmap e79: 8 total, 8 up, 8 in 2024-02-13T15:09:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:23 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]: dispatch 2024-02-13T15:09:24.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]': finished 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[24754]: osdmap e80: 8 total, 8 up, 8 in 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]: dispatch 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]': finished 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[29232]: osdmap e80: 8 total, 8 up, 8 in 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:24.782 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:24 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]: dispatch 2024-02-13T15:09:24.975 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:24 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2241823152' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1844860704"}]': finished 2024-02-13T15:09:24.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:24 smithi098 ceph-mon[25694]: osdmap e80: 8 total, 8 up, 8 in 2024-02-13T15:09:24.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:24 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:24.976 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:24 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]: dispatch 2024-02-13T15:09:26.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-02-13T15:09:26.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]': finished 2024-02-13T15:09:26.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: osdmap e81: 8 total, 8 up, 8 in 2024-02-13T15:09:26.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2298243511' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]': finished 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: osdmap e81: 8 total, 8 up, 8 in 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2298243511' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:26.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-02-13T15:09:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/957831064' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/604203265"}]': finished 2024-02-13T15:09:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: osdmap e81: 8 total, 8 up, 8 in 2024-02-13T15:09:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2298243511' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]: dispatch 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:26.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:09:26.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]': finished 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: osdmap e82: 8 total, 8 up, 8 in 2024-02-13T15:09:27.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3115079472' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]': finished 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: osdmap e82: 8 total, 8 up, 8 in 2024-02-13T15:09:27.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3115079472' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:27.007 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:26 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-13T15:09:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-02-13T15:09:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-13T15:09:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-02-13T15:09:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/291642484"}]': finished 2024-02-13T15:09:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: osdmap e82: 8 total, 8 up, 8 in 2024-02-13T15:09:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3115079472' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:26 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]: dispatch 2024-02-13T15:09:28.010 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[24754]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]': finished 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[24754]: osdmap e83: 8 total, 8 up, 8 in 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/4096319376' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[29232]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]': finished 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[29232]: osdmap e83: 8 total, 8 up, 8 in 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/4096319376' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:28.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:27 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:27 smithi098 ceph-mon[25694]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-02-13T15:09:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:27 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1730337629"}]': finished 2024-02-13T15:09:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:27 smithi098 ceph-mon[25694]: osdmap e83: 8 total, 8 up, 8 in 2024-02-13T15:09:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:27 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/4096319376' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:27 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]: dispatch 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]': finished 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: osdmap e84: 8 total, 8 up, 8 in 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1675599871' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:29.253 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]': finished 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: osdmap e84: 8 total, 8 up, 8 in 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1675599871' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/219490389"}]': finished 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: osdmap e84: 8 total, 8 up, 8 in 2024-02-13T15:09:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1675599871' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]: dispatch 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: Checking dashboard <-> RGW credentials 2024-02-13T15:09:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:09:30.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:30.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:29 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]': finished 2024-02-13T15:09:30.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:29 smithi098 ceph-mon[25694]: osdmap e85: 8 total, 8 up, 8 in 2024-02-13T15:09:30.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:29 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]: dispatch 2024-02-13T15:09:30.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]': finished 2024-02-13T15:09:30.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[24754]: osdmap e85: 8 total, 8 up, 8 in 2024-02-13T15:09:30.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]: dispatch 2024-02-13T15:09:30.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1741139466"}]': finished 2024-02-13T15:09:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[29232]: osdmap e85: 8 total, 8 up, 8 in 2024-02-13T15:09:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:29 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]: dispatch 2024-02-13T15:09:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:31 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]': finished 2024-02-13T15:09:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:31 smithi098 ceph-mon[25694]: osdmap e86: 8 total, 8 up, 8 in 2024-02-13T15:09:31.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:31 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]: dispatch 2024-02-13T15:09:31.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:31 smithi098 ceph-mon[25694]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]': finished 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[24754]: osdmap e86: 8 total, 8 up, 8 in 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]: dispatch 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[24754]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2682606618' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1111087951"}]': finished 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[29232]: osdmap e86: 8 total, 8 up, 8 in 2024-02-13T15:09:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]: dispatch 2024-02-13T15:09:31.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:31 smithi002 ceph-mon[29232]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]': finished 2024-02-13T15:09:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:32 smithi098 ceph-mon[25694]: osdmap e87: 8 total, 8 up, 8 in 2024-02-13T15:09:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:32 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3516523352' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:32 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]': finished 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[29232]: osdmap e87: 8 total, 8 up, 8 in 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3516523352' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1606590937' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1495240999"}]': finished 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[24754]: osdmap e87: 8 total, 8 up, 8 in 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3516523352' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]: dispatch 2024-02-13T15:09:32.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:32] "GET /metrics HTTP/1.1" 200 214902 "" "Prometheus/2.33.4" 2024-02-13T15:09:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:33 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]': finished 2024-02-13T15:09:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:33 smithi098 ceph-mon[25694]: osdmap e88: 8 total, 8 up, 8 in 2024-02-13T15:09:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:33 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3640972038' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:33 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:33 smithi098 ceph-mon[25694]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:33.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]': finished 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[29232]: osdmap e88: 8 total, 8 up, 8 in 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3640972038' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[29232]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3958364651"}]': finished 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[24754]: osdmap e88: 8 total, 8 up, 8 in 2024-02-13T15:09:33.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3640972038' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]: dispatch 2024-02-13T15:09:33.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-mon[24754]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-02-13T15:09:33.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:09:33.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:09:33.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:09:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:09:33.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:09:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:09:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:34 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]': finished 2024-02-13T15:09:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:34 smithi098 ceph-mon[25694]: osdmap e89: 8 total, 8 up, 8 in 2024-02-13T15:09:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:34 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2651064259' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:34 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]': finished 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[29232]: osdmap e89: 8 total, 8 up, 8 in 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2651064259' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1495240999"}]': finished 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[24754]: osdmap e89: 8 total, 8 up, 8 in 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2651064259' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:34 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]: dispatch 2024-02-13T15:09:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:35 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]': finished 2024-02-13T15:09:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:35 smithi098 ceph-mon[25694]: osdmap e90: 8 total, 8 up, 8 in 2024-02-13T15:09:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:35 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1619603093' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:35 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:35 smithi098 ceph-mon[25694]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]': finished 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[29232]: osdmap e90: 8 total, 8 up, 8 in 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1619603093' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[29232]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2721686623"}]': finished 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[24754]: osdmap e90: 8 total, 8 up, 8 in 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1619603093' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]: dispatch 2024-02-13T15:09:35.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:35 smithi002 ceph-mon[24754]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:36 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]': finished 2024-02-13T15:09:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:36 smithi098 ceph-mon[25694]: osdmap e91: 8 total, 8 up, 8 in 2024-02-13T15:09:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:36 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2642213095' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:36.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:36 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]': finished 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[24754]: osdmap e91: 8 total, 8 up, 8 in 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2642213095' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1111087951"}]': finished 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[29232]: osdmap e91: 8 total, 8 up, 8 in 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2642213095' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:36.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:36 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]: dispatch 2024-02-13T15:09:37.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:37 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]': finished 2024-02-13T15:09:37.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:37 smithi098 ceph-mon[25694]: osdmap e92: 8 total, 8 up, 8 in 2024-02-13T15:09:37.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:37 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3735757400' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:37 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:37 smithi098 ceph-mon[25694]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:37.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]': finished 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[24754]: osdmap e92: 8 total, 8 up, 8 in 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3735757400' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[24754]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1741139466"}]': finished 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[29232]: osdmap e92: 8 total, 8 up, 8 in 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3735757400' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]: dispatch 2024-02-13T15:09:37.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:37 smithi002 ceph-mon[29232]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:38 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]': finished 2024-02-13T15:09:38.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:38 smithi098 ceph-mon[25694]: osdmap e93: 8 total, 8 up, 8 in 2024-02-13T15:09:38.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:38 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]: dispatch 2024-02-13T15:09:38.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]': finished 2024-02-13T15:09:38.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[24754]: osdmap e93: 8 total, 8 up, 8 in 2024-02-13T15:09:38.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]: dispatch 2024-02-13T15:09:38.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2705129878"}]': finished 2024-02-13T15:09:38.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[29232]: osdmap e93: 8 total, 8 up, 8 in 2024-02-13T15:09:38.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:38 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]: dispatch 2024-02-13T15:09:39.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:39 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]': finished 2024-02-13T15:09:39.309 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:39 smithi098 ceph-mon[25694]: osdmap e94: 8 total, 8 up, 8 in 2024-02-13T15:09:39.309 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:39 smithi098 ceph-mon[25694]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]': finished 2024-02-13T15:09:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[24754]: osdmap e94: 8 total, 8 up, 8 in 2024-02-13T15:09:39.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[24754]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/504190323' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/760776131"}]': finished 2024-02-13T15:09:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[29232]: osdmap e94: 8 total, 8 up, 8 in 2024-02-13T15:09:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:39 smithi002 ceph-mon[29232]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:09:40.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:41 smithi098 ceph-mon[25694]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T15:09:42.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:41 smithi002 ceph-mon[24754]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T15:09:42.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:41 smithi002 ceph-mon[29232]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T15:09:42.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:42 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:42.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:42] "GET /metrics HTTP/1.1" 200 214934 "" "Prometheus/2.33.4" 2024-02-13T15:09:42.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:42 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:42 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:43.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:43 smithi002 ceph-mon[24754]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-02-13T15:09:43.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:43 smithi002 ceph-mon[29232]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-02-13T15:09:43.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:09:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:09:43.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:09:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:43 smithi098 ceph-mon[25694]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-02-13T15:09:45.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:45 smithi002 ceph-mon[24754]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:45.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:45 smithi002 ceph-mon[29232]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:45.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:45 smithi098 ceph-mon[25694]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:47 smithi098 ceph-mon[25694]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:09:48.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:47 smithi002 ceph-mon[24754]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:09:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:47 smithi002 ceph-mon[29232]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:09:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:49 smithi098 ceph-mon[25694]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-02-13T15:09:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:49 smithi002 ceph-mon[24754]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-02-13T15:09:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:49 smithi002 ceph-mon[29232]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-02-13T15:09:50.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:09:52.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:51 smithi098 ceph-mon[25694]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:52.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:51 smithi002 ceph-mon[24754]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:52.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:51 smithi002 ceph-mon[29232]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:52.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:52 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:52.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:09:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:52] "GET /metrics HTTP/1.1" 200 214934 "" "Prometheus/2.33.4" 2024-02-13T15:09:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:52 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:52 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:09:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:53 smithi098 ceph-mon[25694]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:54.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:53 smithi002 ceph-mon[24754]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:53 smithi002 ceph-mon[29232]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:55.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:55 smithi002 ceph-mon[24754]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:55.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:55 smithi002 ceph-mon[29232]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:55 smithi098 ceph-mon[25694]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:09:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:57 smithi098 ceph-mon[25694]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:58.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:57 smithi002 ceph-mon[24754]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:57 smithi002 ceph-mon[29232]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:09:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:09:59 smithi098 ceph-mon[25694]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:00.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:09:59 smithi002 ceph-mon[29232]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:00.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:09:59 smithi002 ceph-mon[24754]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:00.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:09:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:09:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:00 smithi098 ceph-mon[25694]: overall HEALTH_OK 2024-02-13T15:10:01.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:00 smithi002 ceph-mon[24754]: overall HEALTH_OK 2024-02-13T15:10:01.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:00 smithi002 ceph-mon[29232]: overall HEALTH_OK 2024-02-13T15:10:02.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:01 smithi098 ceph-mon[25694]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:02.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:01 smithi002 ceph-mon[24754]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:02.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:01 smithi002 ceph-mon[29232]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:02 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:02.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:02] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-02-13T15:10:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:02 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:02 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:03 smithi098 ceph-mon[25694]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:04.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:03 smithi002 ceph-mon[29232]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:03 smithi002 ceph-mon[24754]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:05.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:05 smithi002 ceph-mon[24754]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:05.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:05 smithi002 ceph-mon[29232]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:05.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:05 smithi098 ceph-mon[25694]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:08.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:07 smithi098 ceph-mon[25694]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:07 smithi002 ceph-mon[24754]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:07 smithi002 ceph-mon[29232]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:09.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:09 smithi098 ceph-mon[25694]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:09.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:10:09.995 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:09 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:10:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[24754]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:10.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:10:10.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:10:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[29232]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:10:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:09 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:10:10.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:12.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:11 smithi098 ceph-mon[25694]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:12.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:11 smithi002 ceph-mon[24754]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:11 smithi002 ceph-mon[29232]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:12.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:12 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:12.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:12 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:12.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:12] "GET /metrics HTTP/1.1" 200 214818 "" "Prometheus/2.33.4" 2024-02-13T15:10:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:12 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:13 smithi098 ceph-mon[25694]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:13 smithi002 ceph-mon[24754]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:13 smithi002 ceph-mon[29232]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:15.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:15 smithi002 ceph-mon[24754]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:15.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:15 smithi002 ceph-mon[29232]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:15 smithi098 ceph-mon[25694]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:17 smithi098 ceph-mon[25694]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:17 smithi002 ceph-mon[24754]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:18.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:17 smithi002 ceph-mon[29232]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:19.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:19 smithi098 ceph-mon[25694]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:19 smithi002 ceph-mon[24754]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:19 smithi002 ceph-mon[29232]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:19 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:22.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:21 smithi098 ceph-mon[25694]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:22.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:21 smithi002 ceph-mon[24754]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:22.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:21 smithi002 ceph-mon[29232]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:22.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:22 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:22.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:22 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:22.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:22] "GET /metrics HTTP/1.1" 200 214818 "" "Prometheus/2.33.4" 2024-02-13T15:10:23.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:22 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:23 smithi098 ceph-mon[25694]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:24.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:23 smithi002 ceph-mon[29232]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:23 smithi002 ceph-mon[24754]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:25.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:25 smithi002 ceph-mon[24754]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:25.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:25 smithi002 ceph-mon[29232]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:25.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:25 smithi098 ceph-mon[25694]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:27 smithi098 ceph-mon[25694]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:28.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:27 smithi002 ceph-mon[24754]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:27 smithi002 ceph-mon[29232]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:29.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:29 smithi098 ceph-mon[25694]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:29.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:29 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:10:29.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:29 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:10:29.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:29 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:10:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[24754]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[29232]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:10:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:29 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:10:30.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:32.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:31 smithi098 ceph-mon[25694]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:32.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:31 smithi002 ceph-mon[24754]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:32.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:31 smithi002 ceph-mon[29232]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:32.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:32 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:32 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:32.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:32] "GET /metrics HTTP/1.1" 200 214817 "" "Prometheus/2.33.4" 2024-02-13T15:10:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:32 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:33 smithi098 ceph-mon[25694]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:34.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:33 smithi002 ceph-mon[24754]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:34.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:33 smithi002 ceph-mon[29232]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:35.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:35 smithi002 ceph-mon[24754]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:35.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:35 smithi002 ceph-mon[29232]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:35.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:35 smithi098 ceph-mon[25694]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:37 smithi098 ceph-mon[25694]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:38.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:37 smithi002 ceph-mon[24754]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:38.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:37 smithi002 ceph-mon[29232]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:39.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:39 smithi098 ceph-mon[25694]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:40.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:39 smithi002 ceph-mon[24754]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:39 smithi002 ceph-mon[29232]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:40.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:41 smithi098 ceph-mon[25694]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:42.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:41 smithi002 ceph-mon[24754]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:42.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:41 smithi002 ceph-mon[29232]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:42.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:42 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:42.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:42] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-02-13T15:10:42.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:42 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:42 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:43 smithi098 ceph-mon[25694]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:44.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:43 smithi002 ceph-mon[24754]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:44.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:43 smithi002 ceph-mon[29232]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:45.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:45 smithi002 ceph-mon[24754]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:45.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:45 smithi002 ceph-mon[29232]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:45.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:45 smithi098 ceph-mon[25694]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:47 smithi098 ceph-mon[25694]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:48.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:47 smithi002 ceph-mon[24754]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:48.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:47 smithi002 ceph-mon[29232]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:49.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:49 smithi098 ceph-mon[25694]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:49 smithi002 ceph-mon[29232]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:49 smithi002 ceph-mon[24754]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:50.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:10:52.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:51 smithi098 ceph-mon[25694]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:52.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:51 smithi002 ceph-mon[24754]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:51 smithi002 ceph-mon[29232]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:52.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:52 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:52 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:52.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:10:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:52] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-02-13T15:10:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:52 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:10:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:53 smithi098 ceph-mon[25694]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:54.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:53 smithi002 ceph-mon[24754]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:53 smithi002 ceph-mon[29232]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:55 smithi098 ceph-mon[25694]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:55 smithi002 ceph-mon[24754]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:55 smithi002 ceph-mon[29232]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:10:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:57 smithi098 ceph-mon[25694]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:58.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:57 smithi002 ceph-mon[24754]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:58.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:57 smithi002 ceph-mon[29232]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:10:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:10:59 smithi098 ceph-mon[25694]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:00.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:10:59 smithi002 ceph-mon[24754]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:00.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:10:59 smithi002 ceph-mon[29232]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:00.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:10:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:10:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:02.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:01 smithi098 ceph-mon[25694]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:02.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:01 smithi002 ceph-mon[24754]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:02.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:01 smithi002 ceph-mon[29232]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:02 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:02.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:02] "GET /metrics HTTP/1.1" 200 214812 "" "Prometheus/2.33.4" 2024-02-13T15:11:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:02 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:02 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:03 smithi098 ceph-mon[25694]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:03 smithi002 ceph-mon[24754]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:04.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:03 smithi002 ceph-mon[29232]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:05.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:05 smithi098 ceph-mon[25694]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:05 smithi002 ceph-mon[24754]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:05.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:05 smithi002 ceph-mon[29232]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:08.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:07 smithi098 ceph-mon[25694]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:07 smithi002 ceph-mon[24754]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:07 smithi002 ceph-mon[29232]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:11:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:11:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:11:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:11:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:11:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:11:10.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:10.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:09 smithi098 ceph-mon[25694]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:09 smithi002 ceph-mon[24754]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:10.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:09 smithi002 ceph-mon[29232]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:12.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:11 smithi098 ceph-mon[25694]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:12.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:11 smithi002 ceph-mon[24754]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:11 smithi002 ceph-mon[29232]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:12.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:12 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:12.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:12] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-02-13T15:11:12.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:12 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:12 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:13 smithi098 ceph-mon[25694]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:13 smithi002 ceph-mon[24754]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:13 smithi002 ceph-mon[29232]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:15 smithi098 ceph-mon[25694]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:15.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:15 smithi002 ceph-mon[24754]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:15.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:15 smithi002 ceph-mon[29232]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:16.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-13T15:11:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:17 smithi098 ceph-mon[25694]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:17 smithi002 ceph-mon[24754]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:18.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:17 smithi002 ceph-mon[29232]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:19.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-13T15:11:20.011 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:19 smithi002 ceph-mon[24754]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:20.011 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:19 smithi002 ceph-mon[29232]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:20.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:19 smithi098 ceph-mon[25694]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:20.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:19 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:22.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:21 smithi002 ceph-mon[24754]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:22.072 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:21 smithi002 ceph-mon[29232]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:22.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:21 smithi098 ceph-mon[25694]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:22.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-13T15:11:22.749 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:22] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-02-13T15:11:23.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:22 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:22 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:23.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:22 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:23 smithi098 ceph-mon[25694]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:24.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:23 smithi002 ceph-mon[24754]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:24.086 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:23 smithi002 ceph-mon[29232]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:24.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-13T15:11:25.430 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:25 smithi002 ceph-mon[24754]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:25.430 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:25 smithi002 ceph-mon[29232]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:25.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:25 smithi098 ceph-mon[25694]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:26.903 INFO:teuthology.orchestra.run.smithi002.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:27.572 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-13T15:11:27.581 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T15:11:27.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-13T15:11:27.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[29232]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:27.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:27.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:11:27.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:11:27.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[24754]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:11:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:11:27.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:27 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:27 smithi098 ceph-mon[25694]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:27 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:27 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:11:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:27 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:11:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:27 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:29.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[24754]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:29.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[24754]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[24754]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.051 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[29232]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:29.051 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[29232]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.051 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:28 smithi002 ceph-mon[29232]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:28 smithi098 ceph-mon[25694]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:28 smithi098 ceph-mon[25694]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:28 smithi098 ceph-mon[25694]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:11:29.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:29 smithi002 ceph-mon[24754]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:30.031 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:11:30.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:29 smithi002 ceph-mon[29232]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:29 smithi098 ceph-mon[25694]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:30.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 2m ago 3m 16.7M - ba2b418f427c 7104cca414df 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (3m) 2m ago 3m 49.2M - 8.3.5 dad864ee21e9 653965cc4109 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (2m) 2m ago 2m 72.7M - 3.5 e1d6a67b021e 4d82718c3c2e 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443 running (7m) 2m ago 7m 412M - 17.2.0 e1d6a67b021e 2f8444d8e431 2024-02-13T15:11:30.906 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 63c769408f3f 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (9m) 2m ago 9m 47.5M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (8m) 2m ago 8m 42.1M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (8m) 2m ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (3m) 2m ago 3m 13.8M - 1dbe0e931976 74a090108cbe 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (3m) 2m ago 3m 18.0M - 1dbe0e931976 9678aa02bbb9 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (7m) 2m ago 7m 49.6M 4017M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (7m) 2m ago 7m 49.9M 4017M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (6m) 2m ago 6m 46.0M 4017M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (6m) 2m ago 6m 50.3M 4017M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (5m) 2m ago 5m 48.8M 4272M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (5m) 2m ago 5m 45.5M 4272M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:11:30.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (5m) 2m ago 5m 45.0M 4272M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (4m) 2m ago 4m 48.0M 4272M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (2m) 2m ago 4m 44.5M - 514e6a882f6e 00dac66de9fa 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (2m) 2m ago 2m 70.7M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (2m) 2m ago 2m 71.5M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (2m) 2m ago 2m 68.9M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:11:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (2m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:11:31.410 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:11:31.410 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:11:31.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:11:31.411 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:11:31.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:11:31.412 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:11:31.412 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-13T15:11:31.412 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:11:31.412 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:11:31.772 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "", 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 image" 2024-02-13T15:11:31.773 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:11:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:31 smithi098 ceph-mon[25694]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:31 smithi098 ceph-mon[25694]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:31 smithi098 ceph-mon[25694]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:32.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:31 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1125352974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:11:32.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[24754]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[24754]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[24754]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1125352974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[29232]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[29232]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[29232]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:32.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:31 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1125352974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:11:32.305 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:11:32.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:32] "GET /metrics HTTP/1.1" 200 214816 "" "Prometheus/2.33.4" 2024-02-13T15:11:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:32 smithi098 ceph-mon[25694]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:32 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:32 smithi098 ceph-mon[25694]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:32 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1484722987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:11:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[24754]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[24754]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1484722987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[29232]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[29232]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:11:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:32 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1484722987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:11:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:33 smithi098 ceph-mon[25694]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:34.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:33 smithi002 ceph-mon[24754]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:34.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:33 smithi002 ceph-mon[29232]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:35.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:35 smithi098 ceph-mon[25694]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:35.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:35 smithi002 ceph-mon[24754]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:35.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:35 smithi002 ceph-mon[29232]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:37 smithi098 ceph-mon[25694]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:38.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:37 smithi002 ceph-mon[24754]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:38.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:37 smithi002 ceph-mon[29232]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:40.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:39 smithi098 ceph-mon[25694]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:40.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:39 smithi002 ceph-mon[24754]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:39 smithi002 ceph-mon[29232]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:41 smithi098 ceph-mon[25694]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:42.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:41 smithi002 ceph-mon[24754]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:42.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:41 smithi002 ceph-mon[29232]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:42.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:42 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:42.824 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:42] "GET /metrics HTTP/1.1" 200 214795 "" "Prometheus/2.33.4" 2024-02-13T15:11:42.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:42 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:42 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:44.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:43 smithi098 ceph-mon[25694]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:44.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:43 smithi002 ceph-mon[24754]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:44.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:43 smithi002 ceph-mon[29232]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:45.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:45 smithi098 ceph-mon[25694]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:45.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:45 smithi002 ceph-mon[24754]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:45.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:45 smithi002 ceph-mon[29232]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:47 smithi098 ceph-mon[25694]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:48.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:47 smithi002 ceph-mon[24754]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:48.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:47 smithi002 ceph-mon[29232]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:50.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:11:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:49 smithi098 ceph-mon[25694]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:50.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:49 smithi002 ceph-mon[24754]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:49 smithi002 ceph-mon[29232]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: Upgrade: Target is version 19.0.0-1423-ge1b79d5a (unknown) 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, digests ['quay.ceph.io/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f'] 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 on smithi098 2024-02-13T15:11:51.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[24754]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: Upgrade: Target is version 19.0.0-1423-ge1b79d5a (unknown) 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, digests ['quay.ceph.io/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f'] 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 on smithi098 2024-02-13T15:11:51.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:51 smithi002 ceph-mon[29232]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: Upgrade: Target is version 19.0.0-1423-ge1b79d5a (unknown) 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, digests ['quay.ceph.io/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f'] 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 on smithi098 2024-02-13T15:11:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:51 smithi098 ceph-mon[25694]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:52 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:52.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:11:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:52] "GET /metrics HTTP/1.1" 200 214795 "" "Prometheus/2.33.4" 2024-02-13T15:11:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:52 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:52 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:11:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:53 smithi002 ceph-mon[24754]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:53.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:53 smithi002 ceph-mon[29232]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:53 smithi098 ceph-mon[25694]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:55 smithi098 ceph-mon[25694]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:55 smithi002 ceph-mon[24754]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:55.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:55 smithi002 ceph-mon[29232]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:11:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:57 smithi098 ceph-mon[25694]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:58.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:57 smithi002 ceph-mon[24754]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:11:58.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:57 smithi002 ceph-mon[29232]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:00.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:11:59 smithi098 ceph-mon[25694]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:00.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:11:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:11:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:12:00.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:11:59 smithi002 ceph-mon[24754]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:00.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:11:59 smithi002 ceph-mon[29232]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:02.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:01 smithi098 ceph-mon[25694]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:02.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:01 smithi002 ceph-mon[24754]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:02.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:01 smithi002 ceph-mon[29232]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:02.640 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:12:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:02 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:02.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:02] "GET /metrics HTTP/1.1" 200 214759 "" "Prometheus/2.33.4" 2024-02-13T15:12:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:02 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:02 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 2m ago 3m 16.7M - ba2b418f427c 7104cca414df 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (3m) 2m ago 3m 49.2M - 8.3.5 dad864ee21e9 653965cc4109 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (2m) 2m ago 2m 72.7M - 3.5 e1d6a67b021e 4d82718c3c2e 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443 running (8m) 2m ago 8m 412M - 17.2.0 e1d6a67b021e 2f8444d8e431 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 63c769408f3f 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (9m) 2m ago 10m 47.5M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (8m) 2m ago 8m 42.1M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (8m) 2m ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 2m ago 4m 13.8M - 1dbe0e931976 74a090108cbe 2024-02-13T15:12:03.244 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (4m) 2m ago 4m 18.0M - 1dbe0e931976 9678aa02bbb9 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (8m) 2m ago 8m 49.6M 4017M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (7m) 2m ago 7m 49.9M 4017M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (7m) 2m ago 7m 46.0M 4017M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (6m) 2m ago 6m 50.3M 4017M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (6m) 2m ago 6m 48.8M 4272M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (6m) 2m ago 6m 45.5M 4272M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (5m) 2m ago 5m 45.0M 4272M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (5m) 2m ago 5m 48.0M 4272M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (3m) 2m ago 4m 44.5M - 514e6a882f6e 00dac66de9fa 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (2m) 2m ago 2m 70.7M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (2m) 2m ago 2m 71.5M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (2m) 2m ago 2m 68.9M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:12:03.245 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (2m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:12:03.608 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:12:03.609 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:12:03.765 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:03.506Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:03.765 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:03.914 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "0/23 daemons upgraded", 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 image on host smithi098" 2024-02-13T15:12:03.915 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:12:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:03 smithi098 ceph-mon[25694]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:03 smithi098 ceph-mon[25694]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:04.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:03 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/398372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[29232]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[29232]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/398372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[24754]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[24754]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:03 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/398372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:04.275 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:12:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:04 smithi098 ceph-mon[25694]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:04 smithi098 ceph-mon[25694]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:04 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/912148902' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[24754]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[24754]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/912148902' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[29232]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[29232]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:04 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/912148902' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:06.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:05 smithi098 ceph-mon[25694]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:06.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:05 smithi098 ceph-mon[25694]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:06.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:05 smithi002 ceph-mon[24754]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:06.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:05 smithi002 ceph-mon[24754]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:06.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:05 smithi002 ceph-mon[29232]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:06.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:05 smithi002 ceph-mon[29232]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:08.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:07 smithi098 ceph-mon[25694]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:07 smithi002 ceph-mon[24754]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:08.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:07 smithi002 ceph-mon[29232]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:08 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:08 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:08 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:10.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:09 smithi098 ceph-mon[25694]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:10.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[26881]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:12:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:09 smithi002 ceph-mon[24754]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:09 smithi002 ceph-mon[29232]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:12.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:11 smithi098 ceph-mon[25694]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:12.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:11 smithi002 ceph-mon[24754]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:12.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:11 smithi002 ceph-mon[29232]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:12.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:12 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:12.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:12] "GET /metrics HTTP/1.1" 200 214751 "" "Prometheus/2.33.4" 2024-02-13T15:12:12.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:12 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:12 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:13.784 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:12:13.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:13.785 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:13.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:13.785 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:13.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:13 smithi098 ceph-mon[25694]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:13 smithi002 ceph-mon[24754]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:13 smithi002 ceph-mon[29232]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:15.726 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:15 smithi098 ceph-mon[25694]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:15.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:15 smithi002 ceph-mon[24754]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:15.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:15 smithi002 ceph-mon[29232]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:17.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:17 smithi098 ceph-mon[25694]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:17.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:17 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:17.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:17 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:12:17.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:17 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:12:17.958 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:17 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[24754]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[29232]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:12:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:17 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:18.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:18 smithi098 systemd[1]: Stopping Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:12:18.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:18 smithi098 bash[46743]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-13T15:12:18.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:18 smithi098 ceph-mon[25694]: Upgrade: Updating mgr.x 2024-02-13T15:12:18.955 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:18 smithi098 ceph-mon[25694]: Deploying daemon mgr.x on smithi098 2024-02-13T15:12:18.955 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:18 smithi098 podman[46755]: 2024-02-13 15:12:18.639558446 +0000 UTC m=+0.145407383 container died 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, release=754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, name=centos-stream) 2024-02-13T15:12:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:18 smithi002 ceph-mon[24754]: Upgrade: Updating mgr.x 2024-02-13T15:12:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:18 smithi002 ceph-mon[24754]: Deploying daemon mgr.x on smithi098 2024-02-13T15:12:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:18 smithi002 ceph-mon[29232]: Upgrade: Updating mgr.x 2024-02-13T15:12:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:18 smithi002 ceph-mon[29232]: Deploying daemon mgr.x on smithi098 2024-02-13T15:12:19.297 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46755]: 2024-02-13 15:12:19.035880733 +0000 UTC m=+0.541729666 container cleanup 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc.) 2024-02-13T15:12:19.297 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 bash[46755]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x 2024-02-13T15:12:19.297 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 bash[46782]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46769]: 2024-02-13 15:12:19.29585761 +0000 UTC m=+0.646003771 container remove 2f8444d8e431bb9d629d7071b30f70e659a9caa424552da2036dda677b79409f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, release=754, distribution-scope=public, ceph=True, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream) 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 systemd[1]: Stopped Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Consumed 16.389s CPU time. 2024-02-13T15:12:19.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 systemd[1]: Starting Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:12:19.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:19 smithi002 ceph-mon[24754]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:19.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:19 smithi002 ceph-mon[29232]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:19.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:19 smithi098 ceph-mon[25694]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:19.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46885]: 2024-02-13 15:12:19.623275803 +0000 UTC m=+0.017539196 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:12:19.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46885]: 2024-02-13 15:12:19.763983429 +0000 UTC m=+0.158246860 container create ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-13T15:12:19.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46885]: 2024-02-13 15:12:19.983022559 +0000 UTC m=+0.377285944 container init ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-13T15:12:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:19 smithi098 podman[46885]: 2024-02-13 15:12:19.98569525 +0000 UTC m=+0.379958641 container start ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS) 2024-02-13T15:12:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:20 smithi098 ceph-mgr[46904]: -- 172.21.15.98:0/3263630525 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef129161a0 con 0x55ef128ecc00 2024-02-13T15:12:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:20 smithi098 bash[46885]: ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e 2024-02-13T15:12:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:20 smithi098 systemd[1]: Started Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:12:20.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:20 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:20.565+0000 7fc433c7c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:12:20.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:20 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:20.661+0000 7fc433c7c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:12:21.393 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:21.360+0000 7fc433c7c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:12:21.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:21.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:21.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:21.393 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-mon[25694]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:21.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:21.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[29232]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:21.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:21.317Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:12:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:21.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:21 smithi002 ceph-mon[24754]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:21.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:21.460+0000 7fc433c7c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:12:21.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:21.611+0000 7fc433c7c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:12:21.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:21.711+0000 7fc433c7c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:12:22.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:21 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:21.801+0000 7fc433c7c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:12:22.434 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:22.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:22.121+0000 7fc433c7c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:12:22.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:22.234+0000 7fc433c7c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:12:22.434 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:22.326+0000 7fc433c7c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:12:22.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:22 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:22.543 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:22 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:22.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:22.437+0000 7fc433c7c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:12:22.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:22.622+0000 7fc433c7c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:12:22.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:22] "GET /metrics HTTP/1.1" 200 214751 "" "Prometheus/2.33.4" 2024-02-13T15:12:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:23 smithi098 ceph-mon[25694]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:23.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:23 smithi002 ceph-mon[24754]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:23.506 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:23 smithi002 ceph-mon[29232]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:23.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:23 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:12:23.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:12:23.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:23 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:23.506Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:12:23.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:23 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:23.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:24.020 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:23.653+0000 7fc433c7c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:12:24.020 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:23.797+0000 7fc433c7c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:12:24.290 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:24.136+0000 7fc433c7c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:12:24.290 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:24.240+0000 7fc433c7c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:12:24.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:24.450+0000 7fc433c7c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:12:24.904 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:24.649+0000 7fc433c7c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:12:25.235 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:24.957+0000 7fc433c7c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:12:25.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:25.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:25 smithi098 ceph-mon[25694]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:25.542 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:25 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:25.543 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:25.232+0000 7fc433c7c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:12:25.543 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:25.481+0000 7fc433c7c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:12:25.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:25.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[24754]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:25.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:25.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:25.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[29232]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:25.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:25 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:26.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:26.302+0000 7fc433c7c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:12:26.898 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:26.896+0000 7fc433c7c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:12:27.229 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:26.999+0000 7fc433c7c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:12:27.229 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:27.227+0000 7fc433c7c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:12:27.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:27] ENGINE Bus STARTING 2024-02-13T15:12:27.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: CherryPy Checker: 2024-02-13T15:12:27.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: The Application mounted at '' has an empty config. 2024-02-13T15:12:27.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:12:27.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:27] ENGINE Serving on http://:::9283 2024-02-13T15:12:27.583 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:27] ENGINE Bus STARTED 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: Standby manager daemon x restarted 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:12:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:27 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:28.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:28.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: Standby manager daemon x restarted 2024-02-13T15:12:28.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: Standby manager daemon x started 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: Standby manager daemon x restarted 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:28.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:12:28.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:27 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: mgrmap e22: y(active, since 5m), standbys: x 2024-02-13T15:12:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-13T15:12:29.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:28 smithi098 ceph-mon[25694]: osdmap e95: 8 total, 8 up, 8 in 2024-02-13T15:12:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: mgrmap e22: y(active, since 5m), standbys: x 2024-02-13T15:12:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[24754]: osdmap e95: 8 total, 8 up, 8 in 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: mgrmap e22: y(active, since 5m), standbys: x 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-13T15:12:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:28 smithi002 ceph-mon[29232]: osdmap e95: 8 total, 8 up, 8 in 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE Bus STOPPING 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE Bus STOPPED 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE Bus STARTING 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE Serving on http://:::9283 2024-02-13T15:12:29.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:12:29] ENGINE Bus STARTED 2024-02-13T15:12:29.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setuser ceph since I am not root 2024-02-13T15:12:29.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ignoring --setgroup ceph since I am not root 2024-02-13T15:12:29.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mgr[24990]: -- 172.21.15.2:0/888245126 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607bf358340 con 0x5607bf330c00 2024-02-13T15:12:30.067 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:29.662+0000 7fa79e1ab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:12:30.067 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:29.757+0000 7fa79e1ab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:12:30.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:12:30.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: Failing over to other MGR 2024-02-13T15:12:30.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: mgrmap e23: x(active, starting, since 0.995392s) 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:12:30.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: Manager daemon x is now available 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.069 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: Failing over to other MGR 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: mgrmap e23: x(active, starting, since 0.995392s) 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:12:30.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: Manager daemon x is now available 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.072 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:29 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:29] "GET /metrics HTTP/1.1" 200 35134 "" "Prometheus/2.33.4" 2024-02-13T15:12:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-13T15:12:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: Failing over to other MGR 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.14442 172.21.15.2:0/2077519377' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: mgrmap e23: x(active, starting, since 0.995392s) 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:12:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: Manager daemon x is now available 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:29 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-13T15:12:30.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:30.066Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:30.711 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.376+0000 7fa79e1ab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:12:30.711 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.471+0000 7fa79e1ab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:12:30.711 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.557+0000 7fa79e1ab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:12:30.711 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.709+0000 7fa79e1ab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:12:30.992 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.793+0000 7fa79e1ab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:12:30.992 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.905+0000 7fa79e1ab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:12:30.992 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:30.990+0000 7fa79e1ab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:12:30.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: Queued rgw.foo for migration 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: Queued rgw.smpl for migration 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: Deploying cephadm binary to smithi002 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: Deploying cephadm binary to smithi098 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: mgrmap e24: x(active, since 2s) 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: Queued rgw.foo for migration 2024-02-13T15:12:30.993 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: Queued rgw.smpl for migration 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: Deploying cephadm binary to smithi002 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: Deploying cephadm binary to smithi098 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: mgrmap e24: x(active, since 2s) 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:30.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:30 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: Queued rgw.foo for migration 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: Queued rgw.smpl for migration 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: Deploying cephadm binary to smithi002 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: Deploying cephadm binary to smithi098 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: mgrmap e24: x(active, since 2s) 2024-02-13T15:12:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:31.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:30 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:31.247 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:31.158+0000 7fa79e1ab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:12:31.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:31.246+0000 7fa79e1ab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:12:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:31 smithi098 ceph-mon[25694]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:32.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:31 smithi002 ceph-mon[24754]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:32.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:31 smithi002 ceph-mon[29232]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:32.445 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:32.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:12:32.445 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:32.310+0000 7fa79e1ab000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:12:32.445 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:32.443+0000 7fa79e1ab000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:12:32.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:32.639+0000 7fa79e1ab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:12:32.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:32.730+0000 7fa79e1ab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:12:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: [13/Feb/2024:15:12:31] ENGINE Bus STARTING 2024-02-13T15:12:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: [13/Feb/2024:15:12:31] ENGINE Serving on https://172.21.15.98:7150 2024-02-13T15:12:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: [13/Feb/2024:15:12:31] ENGINE Serving on http://172.21.15.98:8765 2024-02-13T15:12:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: [13/Feb/2024:15:12:31] ENGINE Bus STARTED 2024-02-13T15:12:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:32 smithi098 ceph-mon[25694]: mgrmap e25: x(active, since 4s) 2024-02-13T15:12:33.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:32.870+0000 7fa79e1ab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: [13/Feb/2024:15:12:31] ENGINE Bus STARTING 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: [13/Feb/2024:15:12:31] ENGINE Serving on https://172.21.15.98:7150 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: [13/Feb/2024:15:12:31] ENGINE Serving on http://172.21.15.98:8765 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: [13/Feb/2024:15:12:31] ENGINE Bus STARTED 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[24754]: mgrmap e25: x(active, since 4s) 2024-02-13T15:12:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: [13/Feb/2024:15:12:31] ENGINE Bus STARTING 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: [13/Feb/2024:15:12:31] ENGINE Serving on https://172.21.15.98:7150 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: [13/Feb/2024:15:12:31] ENGINE Serving on http://172.21.15.98:8765 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: [13/Feb/2024:15:12:31] ENGINE Bus STARTED 2024-02-13T15:12:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:32 smithi002 ceph-mon[29232]: mgrmap e25: x(active, since 4s) 2024-02-13T15:12:33.507 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:33.198+0000 7fa79e1ab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:12:33.507 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:33.428+0000 7fa79e1ab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:12:33.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:33.516+0000 7fa79e1ab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:12:33.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:33.692+0000 7fa79e1ab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:12:33.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:12:33.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:33.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:33.506Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:12:33.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:33 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:33.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:34.644 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:34 smithi002 ceph-mon[29232]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:34 smithi002 ceph-mon[29232]: mgrmap e26: x(active, since 5s) 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:34 smithi002 ceph-mon[24754]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:34 smithi002 ceph-mon[24754]: mgrmap e26: x(active, since 5s) 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:34 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:34.666+0000 7fa79e1ab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:12:35.031 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:35.029+0000 7fa79e1ab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:12:35.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:34 smithi098 ceph-mon[25694]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:35.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:34 smithi098 ceph-mon[25694]: mgrmap e26: x(active, since 5s) 2024-02-13T15:12:35.303 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:35.140+0000 7fa79e1ab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:12:35.353 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:12:35.353 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 3m ago 4m 16.7M - ba2b418f427c 7104cca414df 2024-02-13T15:12:35.353 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (4m) 11s ago 4m 54.6M - 8.3.5 dad864ee21e9 653965cc4109 2024-02-13T15:12:35.353 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (3m) 3m ago 3m 72.7M - 3.5 e1d6a67b021e 4d82718c3c2e 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283 running (15s) 11s ago 8m 48.2M - 19.0.0-1423-ge1b79d5a 28480620e5ed ce1243ea34d0 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 63c769408f3f 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (10m) 3m ago 10m 47.5M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (9m) 11s ago 9m 43.3M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (9m) 3m ago 9m 40.4M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (5m) 3m ago 5m 13.8M - 1dbe0e931976 74a090108cbe 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (4m) 11s ago 4m 19.8M - 1dbe0e931976 9678aa02bbb9 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (8m) 3m ago 8m 49.6M 4017M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (8m) 3m ago 8m 49.9M 4017M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (7m) 3m ago 7m 46.0M 4017M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (7m) 3m ago 7m 50.3M 4017M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (7m) 11s ago 7m 51.0M 4272M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (6m) 11s ago 6m 49.0M 4272M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (6m) 11s ago 6m 47.4M 4272M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (5m) 11s ago 5m 49.5M 4272M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:12:35.354 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (3m) 11s ago 5m 64.8M - 514e6a882f6e 00dac66de9fa 2024-02-13T15:12:35.355 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (3m) 3m ago 3m 70.7M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:12:35.355 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (3m) 11s ago 3m 72.3M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:12:35.355 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:12:35.355 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (3m) 11s ago 3m 72.0M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:35.301+0000 7fa79e1ab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:12:35] ENGINE Bus STARTING 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: CherryPy Checker: 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: The Application mounted at '' has an empty config. 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:12:35] ENGINE Serving on http://:::9283 2024-02-13T15:12:35.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: [13/Feb/2024:15:12:35] ENGINE Bus STARTED 2024-02-13T15:12:35.765 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:12:35.766 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:12:35.767 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-02-13T15:12:35.767 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:12:35.767 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:12:35.767 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: Standby manager daemon y started 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:35 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1619757445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:36.126 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:35.751Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:36.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: Standby manager daemon y started 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1619757445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: Standby manager daemon y started 2024-02-13T15:12:36.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:12:36.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:12:36.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:12:36.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/2379813985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:12:36.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:35 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1619757445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "1/23 daemons upgraded", 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:12:36.161 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:12:36.581 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:12:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:36 smithi098 ceph-mon[25694]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:36 smithi098 ceph-mon[25694]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:36 smithi098 ceph-mon[25694]: mgrmap e27: x(active, since 7s), standbys: y 2024-02-13T15:12:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:36 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:12:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:36 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/4133393173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:37.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[24754]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[24754]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[24754]: mgrmap e27: x(active, since 7s), standbys: y 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/4133393173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[29232]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[29232]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:37.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[29232]: mgrmap e27: x(active, since 7s), standbys: y 2024-02-13T15:12:37.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:12:37.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:36 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/4133393173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:12:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:37 smithi098 ceph-mon[25694]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:38.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:37 smithi002 ceph-mon[24754]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:38.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:37 smithi002 ceph-mon[29232]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:12:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:38 smithi098 ceph-mon[25694]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:12:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:38 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:38 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:38 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:38 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[24754]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[29232]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:12:39.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:39.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:38 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:40.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:39] "GET /metrics HTTP/1.1" 200 35134 "" "Prometheus/2.33.4" 2024-02-13T15:12:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:12:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi002 to 2737M 2024-02-13T15:12:41.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:40 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi002 to 2737M 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:41.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi002 to 2737M 2024-02-13T15:12:41.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:41.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:40 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:42.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:12:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:12:42.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:12:42.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:42 smithi002 ceph-mon[24754]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:12:42.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:42 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:42.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:12:42.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:42 smithi002 ceph-mon[29232]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:12:42.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:42 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:42 smithi098 ceph-mon[25694]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:12:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:42 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:43.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:12:43.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:43.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:43.877 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:45.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:44 smithi098 ceph-mon[25694]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:45.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:44 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:45.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:44 smithi002 ceph-mon[24754]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:45.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:44 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:45.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:44 smithi002 ceph-mon[29232]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:45.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:44 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:46 smithi098 ceph-mon[25694]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:47.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:46 smithi002 ceph-mon[24754]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:47.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:46 smithi002 ceph-mon[29232]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:49.086 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:48 smithi098 ceph-mon[25694]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:49.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:48 smithi002 ceph-mon[24754]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:48 smithi002 ceph-mon[29232]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:12:50.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:49] "GET /metrics HTTP/1.1" 200 38128 "" "Prometheus/2.33.4" 2024-02-13T15:12:51.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:50 smithi098 ceph-mon[25694]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:51.101 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:50 smithi002 ceph-mon[24754]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:51.101 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:50 smithi002 ceph-mon[29232]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:52.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:12:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:12:52.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:52.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:52 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:12:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:53.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:12:53.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:52 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:12:53.818 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:53 smithi002 systemd[1]: Stopping Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:12:53.818 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:53 smithi002 bash[58278]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-13T15:12:53.818 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=error ts=2024-02-13T15:12:53.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:53.818 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-02-13T15:12:53.818 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=warn ts=2024-02-13T15:12:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-02-13T15:12:54.105 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:53 smithi002 podman[58291]: 2024-02-13 15:12:53.816698295 +0000 UTC m=+0.142125834 container died 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Adjusting osd_memory_target on smithi098 to 2992M 2024-02-13T15:12:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:12:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:12:54.105 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Adjusting osd_memory_target on smithi098 to 2992M 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:12:54.107 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.108 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Adjusting osd_memory_target on smithi098 to 2992M 2024-02-13T15:12:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:12:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:54.356 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 podman[58291]: 2024-02-13 15:12:54.311441159 +0000 UTC m=+0.636868698 container cleanup 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:54.356 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 bash[58291]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a 2024-02-13T15:12:54.356 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 bash[58317]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 podman[58305]: 2024-02-13 15:12:54.604492182 +0000 UTC m=+0.791460259 container remove 74a090108cbeb4b555d83eac091707f7a59936feb5d5964ce5a1c7cfeb527014 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 systemd[1]: Stopped Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 2.656s CPU time. 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 systemd[1]: Starting Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:12:54.865 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 podman[58382]: 2024-02-13 15:12:54.739035664 +0000 UTC m=+0.018353770 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-13T15:12:54.866 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:54 smithi002 podman[58382]: 2024-02-13 15:12:54.864046325 +0000 UTC m=+0.143364418 container create 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:55.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[24754]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:55.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[24754]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-13T15:12:55.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[24754]: Reconfiguring daemon node-exporter.a on smithi002 2024-02-13T15:12:55.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[29232]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:55.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[29232]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-13T15:12:55.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:54 smithi002 ceph-mon[29232]: Reconfiguring daemon node-exporter.a on smithi002 2024-02-13T15:12:55.128 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 podman[58382]: 2024-02-13 15:12:55.098732364 +0000 UTC m=+0.378050458 container init 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:55.128 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 podman[58382]: 2024-02-13 15:12:55.101669505 +0000 UTC m=+0.380987607 container start 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:12:55.128 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.105Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T15:12:55.128 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.105Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T15:12:55.128 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T15:12:55.129 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T15:12:55.130 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T15:12:55.131 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T15:12:55.132 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T15:12:55.133 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a[58396]: ts=2024-02-13T15:12:55.106Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T15:12:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:54 smithi098 ceph-mon[25694]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:54 smithi098 ceph-mon[25694]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-13T15:12:55.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:54 smithi098 ceph-mon[25694]: Reconfiguring daemon node-exporter.a on smithi002 2024-02-13T15:12:55.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 bash[58382]: 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf 2024-02-13T15:12:55.628 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:12:55 smithi002 systemd[1]: Started Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:12:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:56 smithi098 ceph-mon[25694]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:56 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:56 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:56 smithi098 ceph-mon[25694]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:12:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:56 smithi098 ceph-mon[25694]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:12:56.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[24754]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:56.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[24754]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[24754]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[29232]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[29232]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T15:12:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:56 smithi002 ceph-mon[29232]: Reconfiguring daemon alertmanager.a on smithi002 2024-02-13T15:12:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:58 smithi098 ceph-mon[25694]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:58.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:58 smithi002 ceph-mon[24754]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:58.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:58 smithi002 ceph-mon[29232]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:12:59.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:12:59 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:59.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:12:59 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:12:59.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:12:59 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:00.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:12:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:12:59] "GET /metrics HTTP/1.1" 200 38128 "" "Prometheus/2.33.4" 2024-02-13T15:13:00.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:00 smithi098 ceph-mon[25694]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:00 smithi002 ceph-mon[24754]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:00 smithi002 ceph-mon[29232]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:02 smithi098 ceph-mon[25694]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:02 smithi098 ceph-mon[25694]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:02.583 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:02.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:13:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:02 smithi002 ceph-mon[24754]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:02 smithi002 ceph-mon[24754]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:02.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:02 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:13:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:02 smithi002 ceph-mon[29232]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:02 smithi002 ceph-mon[29232]: from='client.14859 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:03.107 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:02 smithi002 systemd[1]: Stopping Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:03.107 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:02 smithi002 bash[58647]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T15:13:03.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[49358]: level=info ts=2024-02-13T15:13:03.104Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:13:03.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 podman[58658]: 2024-02-13 15:13:03.145715331 +0000 UTC m=+0.163056795 container died 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:03.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:03.505Z caller=notifier.go:526 level=error component=notifier alertmanager=http://172.21.15.2:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"http://172.21.15.2:9093/api/v2/alerts\": dial tcp 172.21.15.2:9093: connect: connection refused" 2024-02-13T15:13:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 podman[58658]: 2024-02-13 15:13:03.630347172 +0000 UTC m=+0.647688635 container cleanup 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 bash[58658]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T15:13:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 bash[58684]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T15:13:04.209 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 podman[58672]: 2024-02-13 15:13:03.881722351 +0000 UTC m=+0.736152838 container remove 7104cca414dfdb36e89b61a593d78f256025cd9e298ef4ef5bfa79e91e35e00c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:04.209 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 podman[58672]: 2024-02-13 15:13:03.966043442 +0000 UTC m=+0.820473889 volume remove a881718371aa375f46d32d52cacdd99e07e615061d18f7a9f4c5947d62fa763c 2024-02-13T15:13:04.209 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-13T15:13:04.209 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 systemd[1]: Stopped Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:04.209 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:03 smithi002 systemd[1]: Starting Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:04.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-mon[24754]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:04.469 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-mon[29232]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:04.469 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 podman[58754]: 2024-02-13 15:13:04.208362636 +0000 UTC m=+0.115207184 volume create d1abf9e8a5d45a2244f72e6a708ccdb275bef79f10e5060dac2ebd3d8f16be55 2024-02-13T15:13:04.469 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 podman[58754]: 2024-02-13 15:13:04.111485505 +0000 UTC m=+0.018330053 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-13T15:13:04.470 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 podman[58754]: 2024-02-13 15:13:04.467132067 +0000 UTC m=+0.373976616 container create a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:04 smithi098 ceph-mon[25694]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 podman[58754]: 2024-02-13 15:13:04.786591933 +0000 UTC m=+0.693436485 container init a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 podman[58754]: 2024-02-13 15:13:04.789383538 +0000 UTC m=+0.696228086 container start a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.799Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.800Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.801Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.802Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T15:13:05.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.828Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:13:05.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.828Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T15:13:05.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.830Z caller=main.go:518 msg=Listening address=:9093 2024-02-13T15:13:05.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:04.830Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-13T15:13:05.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 bash[58754]: a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf 2024-02-13T15:13:05.128 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:04 smithi002 systemd[1]: Started Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:06.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[24754]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:06.205 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:05 smithi002 ceph-mon[29232]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:05 smithi098 ceph-mon[25694]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:06.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:06.803Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001061823s 2024-02-13T15:13:06.952 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:13:07.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:06 smithi098 ceph-mon[25694]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:07.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:06 smithi002 ceph-mon[24754]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:07.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:06 smithi002 ceph-mon[29232]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:07.635 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:13:07.635 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 starting - - - - 2024-02-13T15:13:07.635 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (4m) 28s ago 4m 55.1M - 8.3.5 dad864ee21e9 653965cc4109 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (3m) 29s ago 3m 51.0M - 3.5 e1d6a67b021e 4d82718c3c2e 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283 running (48s) 28s ago 9m 489M - 19.0.0-1423-ge1b79d5a 28480620e5ed ce1243ea34d0 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (11m) 29s ago 11m 380M - 17.2.0 e1d6a67b021e 63c769408f3f 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (11m) 29s ago 11m 54.2M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (9m) 28s ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (9m) 29s ago 9m 43.7M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 starting - - - - 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (5m) 28s ago 5m 17.7M - 1.3.1 1dbe0e931976 9678aa02bbb9 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (9m) 29s ago 9m 50.8M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (8m) 29s ago 8m 52.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (8m) 29s ago 8m 47.5M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (7m) 29s ago 7m 52.2M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (7m) 28s ago 7m 51.5M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:13:07.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (7m) 28s ago 7m 49.2M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (6m) 28s ago 6m 48.0M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (6m) 28s ago 6m 50.1M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (4m) 28s ago 5m 64.9M - 2.33.4 514e6a882f6e 00dac66de9fa 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (4m) 29s ago 4m 72.0M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (4m) 28s ago 4m 72.8M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (3m) 29s ago 3m 70.2M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:13:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (3m) 28s ago 3m 72.8M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-13T15:13:08.051 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:13:08.052 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:13:08.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:07 smithi002 ceph-mon[24754]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:08.053 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:07 smithi002 ceph-mon[29232]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:07 smithi098 ceph-mon[25694]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:08.392 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:13:08.392 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:13:08.392 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:13:08.392 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:13:08.393 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-13T15:13:08.393 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "1/23 daemons upgraded", 2024-02-13T15:13:08.393 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-02-13T15:13:08.393 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:13:08.393 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:13:08.805 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:13:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-mon[25694]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-mon[25694]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-mon[25694]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2939527536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/903018883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:13:08.741+0000 7fc3b9973700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Creating ceph-iscsi config... 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:09.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return _run_code(code, main_globals, None, 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: exec(code, run_globals) 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:09.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return future.result(timeout) 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return self.__get_result() 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: raise self._exception 2024-02-13T15:13:09.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: config_blobs=daemon_spec.final_config, 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Creating ceph-iscsi config... 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return _run_code(code, main_globals, None, 2024-02-13T15:13:09.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: exec(code, run_globals) 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:09.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:09.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[24754]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[24754]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[24754]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2939527536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:09.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/903018883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:09.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[29232]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[29232]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[29232]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2939527536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:09.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:08 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/903018883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:10.333 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:09] "GET /metrics HTTP/1.1" 200 38128 "" "Prometheus/2.33.4" 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:09 smithi098 ceph-mon[25694]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:09 smithi098 ceph-mon[25694]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.334 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: exec(code, run_globals) 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.335 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:13:10.336 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: return future.result(timeout) 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: return self.__get_result() 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: raise self._exception 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:13:10.337 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.338 INFO:journalctl@ceph.mon.b.smithi098.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: exec(code, run_globals) 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.339 INFO:journalctl@ceph.mon.b.smithi098.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:09 smithi098 ceph-mon[25694]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:13:10.340 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:09 smithi098 ceph-mon[25694]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:13:10.380 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[24754]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:10.381 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[24754]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.381 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.381 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.381 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.381 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.382 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:13:10.383 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:13:10.384 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.385 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.386 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[29232]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[29232]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.387 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.388 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:13:10.389 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:13:10.390 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:13:10.391 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.392 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[24754]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[24754]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[29232]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:13:10.393 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:09 smithi002 ceph-mon[29232]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:13:11.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:10 smithi098 ceph-mon[25694]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:11.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:10 smithi002 ceph-mon[24754]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:11.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:10 smithi002 ceph-mon[29232]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:12.583 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:13:12.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:13:13.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:12 smithi098 ceph-mon[25694]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:13.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:12 smithi002 ceph-mon[24754]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:13.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:12 smithi002 ceph-mon[29232]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:14 smithi002 ceph-mon[24754]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:14 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:15.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:13:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:13:14.806Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003810053s 2024-02-13T15:13:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:14 smithi002 ceph-mon[29232]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:14 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:15.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:14 smithi098 ceph-mon[25694]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:15.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:14 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:16 smithi098 ceph-mon[25694]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:17.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:16 smithi002 ceph-mon[24754]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:17.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:16 smithi002 ceph-mon[29232]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:19.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:18 smithi002 ceph-mon[24754]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:19.142 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:18 smithi002 ceph-mon[29232]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:18 smithi098 ceph-mon[25694]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:13:20.222 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:19 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:19] "GET /metrics HTTP/1.1" 200 38124 "" "Prometheus/2.33.4" 2024-02-13T15:13:21.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:20 smithi098 ceph-mon[25694]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:13:21.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:20 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2084338172' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:13:21.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:20 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2535291720' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:21.208 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:20 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:21.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[24754]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:13:21.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2084338172' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:13:21.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2535291720' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:21.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:21.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[29232]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:13:21.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2084338172' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:13:21.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2535291720' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:21.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:20 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]: dispatch 2024-02-13T15:13:22.315 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:21 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]': finished 2024-02-13T15:13:22.315 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:21 smithi098 ceph-mon[25694]: osdmap e96: 8 total, 8 up, 8 in 2024-02-13T15:13:22.315 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:21 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1390214255' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.315 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:21 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]': finished 2024-02-13T15:13:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[24754]: osdmap e96: 8 total, 8 up, 8 in 2024-02-13T15:13:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1390214255' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2481248937"}]': finished 2024-02-13T15:13:22.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[29232]: osdmap e96: 8 total, 8 up, 8 in 2024-02-13T15:13:22.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1390214255' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:21 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]: dispatch 2024-02-13T15:13:22.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:13:22.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 systemd[1]: Stopping Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:22.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[24986]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 bash[50391]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-13T15:13:22.964 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.763Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.764Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.764Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.764Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[42605]: ts=2024-02-13T15:13:22.764Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-13T15:13:22.965 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:22 smithi098 podman[50404]: 2024-02-13 15:13:22.773984256 +0000 UTC m=+0.173360999 container died 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:23.332 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50404]: 2024-02-13 15:13:23.105410139 +0000 UTC m=+0.504786887 container cleanup 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:23.332 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 bash[50404]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T15:13:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-mon[25694]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-13T15:13:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]': finished 2024-02-13T15:13:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-mon[25694]: osdmap e97: 8 total, 8 up, 8 in 2024-02-13T15:13:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1279245779' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:22 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[29232]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]': finished 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[29232]: osdmap e97: 8 total, 8 up, 8 in 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1279245779' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[24754]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3497843875"}]': finished 2024-02-13T15:13:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[24754]: osdmap e97: 8 total, 8 up, 8 in 2024-02-13T15:13:23.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1279245779' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:22 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]: dispatch 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50418]: 2024-02-13 15:13:23.366948753 +0000 UTC m=+0.595460502 container remove 00dac66de9fafed9a023b853a10bd240a71d9e56ac6f522c9b8826b4b303d0ba (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 bash[50431]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 systemd[1]: Stopped Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.471s CPU time. 2024-02-13T15:13:23.726 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 systemd[1]: Starting Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:23.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]': finished 2024-02-13T15:13:23.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-mon[25694]: osdmap e98: 8 total, 8 up, 8 in 2024-02-13T15:13:23.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3418363998' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:23.985 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:23.985 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50497]: 2024-02-13 15:13:23.628958805 +0000 UTC m=+0.027029186 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-13T15:13:23.985 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50497]: 2024-02-13 15:13:23.733202187 +0000 UTC m=+0.131272546 container create f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:23.985 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50497]: 2024-02-13 15:13:23.95240083 +0000 UTC m=+0.350471194 container init f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:23.985 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 podman[50497]: 2024-02-13 15:13:23.955083677 +0000 UTC m=+0.353154037 container start f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:13:24.308 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.983Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-13T15:13:24.308 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.984Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-13T15:13:24.308 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.984Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-13T15:13:24.308 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.984Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi098 (none))" 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.984Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.984Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.985Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.986Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.987Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.990Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.990Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.745µs 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:23.990Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 bash[50497]: f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 systemd[1]: Started Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:24.309 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.185Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-02-13T15:13:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]': finished 2024-02-13T15:13:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[24754]: osdmap e98: 8 total, 8 up, 8 in 2024-02-13T15:13:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/3418363998' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1838017178"}]': finished 2024-02-13T15:13:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[29232]: osdmap e98: 8 total, 8 up, 8 in 2024-02-13T15:13:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3418363998' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:23 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]: dispatch 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.472Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.472Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.472Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.446µs wal_replay_duration=482.54388ms total_replay_duration=482.586312ms 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.475Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.475Z caller=main.go:947 level=info msg="TSDB started" 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.475Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.504Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.565449ms db_storage=871ns remote_storage=1.605µs web_handler=389ns query_engine=926ns scrape=555.788µs scrape_sd=167.269µs notify=18.484µs notify_sd=17.584µs rules=27.4475ms 2024-02-13T15:13:24.582 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:24.504Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-13T15:13:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T15:13:24.974 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]': finished 2024-02-13T15:13:24.974 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:24 smithi098 systemd[1]: Stopping Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:24.974 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:24 smithi098 bash[50628]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-13T15:13:24.975 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:24 smithi098 podman[50641]: 2024-02-13 15:13:24.817889009 +0000 UTC m=+0.133261812 container died 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:25.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: osdmap e99: 8 total, 8 up, 8 in 2024-02-13T15:13:25.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.252 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:24 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]: dispatch 2024-02-13T15:13:25.252 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50641]: 2024-02-13 15:13:25.16519919 +0000 UTC m=+0.480571993 container cleanup 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:25.252 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 bash[50641]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b 2024-02-13T15:13:25.252 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 bash[50669]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]': finished 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: osdmap e99: 8 total, 8 up, 8 in 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]: dispatch 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2833814139"}]': finished 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: osdmap e99: 8 total, 8 up, 8 in 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:24 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]: dispatch 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50657]: 2024-02-13 15:13:25.458574521 +0000 UTC m=+0.642349955 container remove 9678aa02bbb984138fe7f30eaa6214023a0052bc26b209af0af9b6a8884337f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 systemd[1]: Stopped Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 2.739s CPU time. 2024-02-13T15:13:25.582 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 systemd[1]: Starting Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:25.948 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50738]: 2024-02-13 15:13:25.592733487 +0000 UTC m=+0.017490797 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-13T15:13:25.948 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50738]: 2024-02-13 15:13:25.709779114 +0000 UTC m=+0.134536452 container create 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:25.948 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50738]: 2024-02-13 15:13:25.947434793 +0000 UTC m=+0.372192098 container init 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-mon[25694]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-13T15:13:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-mon[25694]: Reconfiguring daemon node-exporter.b on smithi098 2024-02-13T15:13:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]': finished 2024-02-13T15:13:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-mon[25694]: osdmap e100: 8 total, 8 up, 8 in 2024-02-13T15:13:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]: dispatch 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 podman[50738]: 2024-02-13 15:13:25.950439902 +0000 UTC m=+0.375197199 container start 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.954Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.954Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.954Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.954Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T15:13:26.333 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T15:13:26.334 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T15:13:26.335 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T15:13:26.336 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T15:13:26.363 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T15:13:26.363 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T15:13:26.363 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b[50752]: ts=2024-02-13T15:13:25.955Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T15:13:26.363 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:26 smithi098 bash[50738]: 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 2024-02-13T15:13:26.363 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:13:26 smithi098 systemd[1]: Started Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[29232]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[29232]: Reconfiguring daemon node-exporter.b on smithi098 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]': finished 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[29232]: osdmap e100: 8 total, 8 up, 8 in 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]: dispatch 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[24754]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-13T15:13:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[24754]: Reconfiguring daemon node-exporter.b on smithi098 2024-02-13T15:13:26.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/718263217' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3088002110"}]': finished 2024-02-13T15:13:26.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[24754]: osdmap e100: 8 total, 8 up, 8 in 2024-02-13T15:13:26.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:25 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]: dispatch 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]': finished 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: osdmap e101: 8 total, 8 up, 8 in 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:27.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/74392136' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:27.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:26 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:27.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:13:27.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]': finished 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: osdmap e101: 8 total, 8 up, 8 in 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/74392136' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:27.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/470498173' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3088002110"}]': finished 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: osdmap e101: 8 total, 8 up, 8 in 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:27.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:27.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/74392136' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:27.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:26 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]: dispatch 2024-02-13T15:13:28.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: Reconfiguring grafana.a (dependencies changed)... 2024-02-13T15:13:28.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T15:13:28.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:28.280 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: Reconfiguring daemon grafana.a on smithi098 2024-02-13T15:13:28.280 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]': finished 2024-02-13T15:13:28.280 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:27 smithi098 ceph-mon[25694]: osdmap e102: 8 total, 8 up, 8 in 2024-02-13T15:13:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: Reconfiguring grafana.a (dependencies changed)... 2024-02-13T15:13:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: Reconfiguring daemon grafana.a on smithi098 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]': finished 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[24754]: osdmap e102: 8 total, 8 up, 8 in 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: Reconfiguring grafana.a (dependencies changed)... 2024-02-13T15:13:28.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T15:13:28.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T15:13:28.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: Reconfiguring daemon grafana.a on smithi098 2024-02-13T15:13:28.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/431916306"}]': finished 2024-02-13T15:13:28.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:27 smithi002 ceph-mon[29232]: osdmap e102: 8 total, 8 up, 8 in 2024-02-13T15:13:29.289 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:28 smithi098 ceph-mon[25694]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:28 smithi002 ceph-mon[24754]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:28 smithi002 ceph-mon[29232]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:30.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:29 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:30.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:29 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:30.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:29 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:31.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:30 smithi098 ceph-mon[25694]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-13T15:13:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:30 smithi002 ceph-mon[24754]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-13T15:13:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:30 smithi002 ceph-mon[29232]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-13T15:13:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:31 smithi098 ceph-mon[25694]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:31 smithi002 ceph-mon[24754]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:31 smithi002 ceph-mon[29232]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:32 smithi098 ceph-mon[25694]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:13:33.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:32 smithi002 ceph-mon[24754]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:13:33.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:32 smithi002 ceph-mon[29232]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:13:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:34 smithi098 ceph-mon[25694]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:34 smithi002 ceph-mon[24754]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:34 smithi002 ceph-mon[29232]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:13:37.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:36 smithi098 ceph-mon[25694]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s 2024-02-13T15:13:37.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:36 smithi002 ceph-mon[24754]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s 2024-02-13T15:13:37.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:36 smithi002 ceph-mon[29232]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s 2024-02-13T15:13:38.660 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:38] "GET /metrics HTTP/1.1" 200 38127 "" "Prometheus/2.33.4" 2024-02-13T15:13:39.000 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi098:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:13:39.175 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:13:39.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:38 smithi098 ceph-mon[25694]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:13:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:38 smithi002 ceph-mon[24754]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:13:39.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:38 smithi002 ceph-mon[29232]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 starting - - - - 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (5m) 61s ago 5m 55.1M - 8.3.5 dad864ee21e9 653965cc4109 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (4m) 61s ago 4m 51.0M - 3.5 e1d6a67b021e 4d82718c3c2e 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283 running (80s) 61s ago 9m 489M - 19.0.0-1423-ge1b79d5a 28480620e5ed ce1243ea34d0 2024-02-13T15:13:39.840 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (11m) 61s ago 11m 380M - 17.2.0 e1d6a67b021e 63c769408f3f 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (11m) 61s ago 11m 54.2M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (10m) 61s ago 10m 46.5M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (10m) 61s ago 10m 43.7M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 starting - - - - 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 starting - - - - 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (9m) 61s ago 9m 50.8M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (9m) 61s ago 9m 52.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (8m) 61s ago 8m 47.5M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (8m) 61s ago 8m 52.2M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (8m) 61s ago 8m 51.5M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:13:39.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (7m) 61s ago 7m 49.2M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (7m) 61s ago 7m 48.0M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (6m) 61s ago 6m 50.1M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 starting - - - - 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (4m) 61s ago 4m 72.0M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (4m) 61s ago 4m 72.8M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (4m) 61s ago 4m 70.2M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:13:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (4m) 61s ago 4m 72.8M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:13:39.929 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:39 smithi098 systemd[1]: Stopping Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:39.929 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:39 smithi098 bash[50945]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-13T15:13:40.205 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[41811]: t=2024-02-13T15:13:39+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-13T15:13:40.205 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:39 smithi098 podman[50958]: 2024-02-13 15:13:39.93802812 +0000 UTC m=+0.125507461 container died 653965cc4109f63b9db151bdae8f2246ff9a32facf8ab11f5212754343da18ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, build-date=2022-03-28T10:36:18.413762, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, io.openshift.expose-services=, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:13:40.244 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-13T15:13:40.245 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:13:40.246 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:13:40.246 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:13:40.576 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 podman[50958]: 2024-02-13 15:13:40.28605991 +0000 UTC m=+0.473539253 container cleanup 653965cc4109f63b9db151bdae8f2246ff9a32facf8ab11f5212754343da18ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, vendor=Red Hat, Inc., version=8.5, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-type=git, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.expose-services=) 2024-02-13T15:13:40.576 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 bash[50958]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a 2024-02-13T15:13:40.576 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 bash[50985]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-13T15:13:40.583 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "1/23 daemons upgraded", 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:13:40.832 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 podman[50972]: 2024-02-13 15:13:40.574365326 +0000 UTC m=+0.635940720 container remove 653965cc4109f63b9db151bdae8f2246ff9a32facf8ab11f5212754343da18ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base rhel8, vcs-type=git, name=ubi8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182) 2024-02-13T15:13:40.832 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-13T15:13:40.832 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 systemd[1]: Stopped Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:40.832 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@grafana.a.service: Consumed 2.566s CPU time. 2024-02-13T15:13:40.833 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 systemd[1]: Starting Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:40.833 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 podman[51055]: 2024-02-13 15:13:40.709017628 +0000 UTC m=+0.018361883 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-13T15:13:40.833 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:40 smithi098 podman[51055]: 2024-02-13 15:13:40.822238007 +0000 UTC m=+0.131582266 container create f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, architecture=x86_64, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.openshift.expose-services=, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-02-13T15:13:40.988 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:13:41.232 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-02-13T15:13:41.232 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.232 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1354911751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:41.233 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/2703733635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:41.233 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 podman[51055]: 2024-02-13 15:13:41.074939827 +0000 UTC m=+0.384284077 container init f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8.5, name=ubi8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-13T15:13:41.233 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 podman[51055]: 2024-02-13 15:13:41.077642868 +0000 UTC m=+0.386987126 container start f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, description=Ceph Grafana Container, vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762) 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-13T15:13:41.234 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="App mode production" logger=settings 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-13T15:13:41.235 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=1.006634ms 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 bash[51055]: f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 systemd[1]: Started Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-13T15:13:41.236 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-13T15:13:41.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-02-13T15:13:41.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1354911751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/2703733635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:41.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1354911751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:13:41.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:41 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/2703733635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:13:41.579 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-02-13T15:13:41.579 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-13T15:13:41.580 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-02-13T15:13:41.580 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-13T15:13:41.580 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-13T15:13:41.580 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:13:41+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPING 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPED 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTING 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Serving on http://:::9283 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTED 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPING 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPED 2024-02-13T15:13:41.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTING 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Serving on http://:::9283 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTED 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPING 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STOPPED 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTING 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Serving on http://:::9283 2024-02-13T15:13:41.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:13:41] ENGINE Bus STARTED 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:42.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:42.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:42.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:42 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:13:42.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:42.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:13:42.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:42.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:42 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:13:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:43.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:43.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:43 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi098.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi098.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:43.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:43 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:13:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: Upgrade: Updating mgr.y 2024-02-13T15:13:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:13:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: Deploying daemon mgr.y on smithi002 2024-02-13T15:13:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:44 smithi098 ceph-mon[25694]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: Upgrade: Updating mgr.y 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: Deploying daemon mgr.y on smithi002 2024-02-13T15:13:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[24754]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: Upgrade: Updating mgr.y 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: Deploying daemon mgr.y on smithi002 2024-02-13T15:13:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:44 smithi002 ceph-mon[29232]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:45 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:45.371 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:45 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:45.372 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:45 smithi002 systemd[1]: Stopping Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:45.372 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:45 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:13:45.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:45 smithi002 podman[60018]: 2024-02-13 15:13:45.370370547 +0000 UTC m=+0.138031947 container died 63c769408f3f3ed37745be68fca8e7a564586bd8b181f6bf100544d91bd60c86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream) 2024-02-13T15:13:46.057 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:45 smithi002 podman[60018]: 2024-02-13 15:13:45.747822744 +0000 UTC m=+0.515484143 container cleanup 63c769408f3f3ed37745be68fca8e7a564586bd8b181f6bf100544d91bd60c86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-02-13T15:13:46.058 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:45 smithi002 bash[60018]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y 2024-02-13T15:13:46.319 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:46 smithi002 ceph-mon[24754]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:46.319 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:46 smithi002 ceph-mon[29232]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:46.320 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 podman[60032]: 2024-02-13 15:13:46.132745658 +0000 UTC m=+0.762477749 container remove 63c769408f3f3ed37745be68fca8e7a564586bd8b181f6bf100544d91bd60c86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, architecture=x86_64, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0) 2024-02-13T15:13:46.320 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:13:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:46 smithi098 ceph-mon[25694]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:46.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-13T15:13:46.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 systemd[1]: Stopped Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:46.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service: Consumed 56.389s CPU time. 2024-02-13T15:13:46.627 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 systemd[1]: Starting Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:13:47.037 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 podman[60162]: 2024-02-13 15:13:46.64297985 +0000 UTC m=+0.018591784 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:13:47.037 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:46 smithi002 podman[60162]: 2024-02-13 15:13:46.742293103 +0000 UTC m=+0.117905031 container create 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:13:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 podman[60162]: 2024-02-13 15:13:47.036426673 +0000 UTC m=+0.412038602 container init 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-02-13T15:13:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 podman[60162]: 2024-02-13 15:13:47.039201985 +0000 UTC m=+0.414813915 container start 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-02-13T15:13:47.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 ceph-mgr[60181]: -- 172.21.15.2:0/1167095963 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555da48161a0 con 0x555da47ecc00 2024-02-13T15:13:47.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 bash[60162]: 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 2024-02-13T15:13:47.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 systemd[1]: Started Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:13:47.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:47.611+0000 7fd5e9946200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:13:47.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:47.712+0000 7fd5e9946200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:13:48.522 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:48.424+0000 7fd5e9946200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:13:48.522 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:48.522 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.522 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:48] "GET /metrics HTTP/1.1" 200 38118 "" "Prometheus/2.33.4" 2024-02-13T15:13:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:48.772 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:48.519+0000 7fd5e9946200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:13:48.772 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:48.668+0000 7fd5e9946200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:13:48.927 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi098:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:13:49.071 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:48.769+0000 7fd5e9946200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:13:49.071 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:48 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:48.860+0000 7fd5e9946200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:13:49.347 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:49.152+0000 7fd5e9946200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:13:49.347 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:49.274+0000 7fd5e9946200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:13:49.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:49.368+0000 7fd5e9946200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:13:49.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:49.481+0000 7fd5e9946200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:13:50.097 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:49.664+0000 7fd5e9946200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:13:50.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:50 smithi002 ceph-mon[24754]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:50.759 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:50 smithi002 ceph-mon[29232]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:50 smithi098 ceph-mon[25694]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:51.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:50 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:50.756+0000 7fd5e9946200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:13:51.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:50 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:50.897+0000 7fd5e9946200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:13:51.417 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:51.254+0000 7fd5e9946200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:13:51.417 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:51.350+0000 7fd5e9946200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:13:51.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-mon[24754]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:51.776 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:51.558+0000 7fd5e9946200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:13:51.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-mon[29232]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:51.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:51 smithi098 ceph-mon[25694]: from='client.15213 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:13:52.081 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:51.773+0000 7fd5e9946200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:13:52.348 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:52.079+0000 7fd5e9946200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:13:52.348 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:52.346+0000 7fd5e9946200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:13:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:52 smithi002 ceph-mon[24754]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:52.609 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:52 smithi002 ceph-mon[29232]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:52 smithi098 ceph-mon[25694]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:52.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:52.606+0000 7fd5e9946200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:13:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:53 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:53.482+0000 7fd5e9946200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:13:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:53 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.374 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:54.104+0000 7fd5e9946200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:13:54.374 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:54.218+0000 7fd5e9946200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:13:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: Standby manager daemon y restarted 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: Standby manager daemon y started 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: Standby manager daemon y restarted 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: Standby manager daemon y started 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:54.457+0000 7fd5e9946200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:13:54] ENGINE Bus STARTING 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: CherryPy Checker: 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: The Application mounted at '' has an empty config. 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:13:54] ENGINE Serving on http://:::9283 2024-02-13T15:13:54.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:13:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:13:54] ENGINE Bus STARTED 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: Standby manager daemon y restarted 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: Standby manager daemon y started 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:13:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-13T15:13:54.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:54 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:13:56.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:56 smithi002 ceph-mon[24754]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:56.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:56 smithi002 ceph-mon[24754]: mgrmap e28: x(active, since 87s), standbys: y 2024-02-13T15:13:56.690 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:56 smithi002 ceph-mon[29232]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:56.690 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:56 smithi002 ceph-mon[29232]: mgrmap e28: x(active, since 87s), standbys: y 2024-02-13T15:13:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:56 smithi098 ceph-mon[25694]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:13:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:56 smithi098 ceph-mon[25694]: mgrmap e28: x(active, since 87s), standbys: y 2024-02-13T15:13:58.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:13:58] "GET /metrics HTTP/1.1" 200 38118 "" "Prometheus/2.33.4" 2024-02-13T15:13:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-mon[25694]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:58.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[24754]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: Reconfiguring iscsi.foo.smithi002.fozvhf (dependencies changed)... 2024-02-13T15:13:58.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:13:58.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:13:58.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:13:58 smithi002 ceph-mon[29232]: Reconfiguring daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:13:59.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:13:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:13:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi098:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:14:00.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-mon[25694]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:00.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:00.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:00 smithi002 ceph-mon[24754]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:00.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:00 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:00.593 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:00 smithi002 ceph-mon[29232]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:00.593 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:00 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:00.892+0000 7fc3b9973700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Creating ceph-iscsi config... 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.083 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: exec(code, run_globals) 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:14:01.084 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return future.result(timeout) 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return self.__get_result() 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: raise self._exception 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: config_blobs=daemon_spec.final_config, 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Creating ceph-iscsi config... 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.085 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: Traceback (most recent call last): 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: exec(code, run_globals) 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.086 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.087 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.484 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:01.225+0000 7fc42727c700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-13T15:14:01.484 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ignoring --setuser ceph since I am not root 2024-02-13T15:14:01.484 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ignoring --setgroup ceph since I am not root 2024-02-13T15:14:01.484 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mgr[46904]: -- 172.21.15.98:0/601085640 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55769b4ee1a0 con 0x55769b4c4c00 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.485 INFO:journalctl@ceph.mon.b.smithi098.stdout: exec(code, run_globals) 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:14:01.486 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: return future.result(timeout) 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: return self.__get_result() 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: raise self._exception 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.487 INFO:journalctl@ceph.mon.b.smithi098.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: exec(code, run_globals) 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.488 INFO:journalctl@ceph.mon.b.smithi098.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: Failing over to other MGR 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.489 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-mon[25694]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:01.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.498 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:14:01.500 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.501 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1102, in _check_daemons 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2389, in _daemon_action 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 769, in wait_async 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-02-13T15:14:01.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1362, in _create_daemon 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: config_blobs=daemon_spec.final_config, 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1664, in _run_cephadm 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:14:01.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE Bus STOPPING 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE Bus STOPPED 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE Bus STARTING 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE Serving on http://:::9283 2024-02-13T15:14:01.506 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:01] ENGINE Bus STARTED 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: Failing over to other MGR 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[24754]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1072, in deploy_daemon 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: Failing over to other MGR 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: from='mgr.24878 172.21.15.98:0/2745526256' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-13T15:14:01.508 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:01 smithi002 ceph-mon[29232]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:02.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:01.801+0000 7f003150d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:14:02.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:01.897+0000 7f003150d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:14:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.24878 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-13T15:14:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: mgrmap e29: y(active, starting, since 0.323744s) 2024-02-13T15:14:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: Manager daemon y is now available 2024-02-13T15:14:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.24878 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: mgrmap e29: y(active, starting, since 0.323744s) 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:02.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: Manager daemon y is now available 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.24878 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-13T15:14:02.598 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: mgrmap e29: y(active, starting, since 0.323744s) 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:02.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:02.600 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: Manager daemon y is now available 2024-02-13T15:14:02.600 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:02.600 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:02.600 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:02.600 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:02 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:02.932 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:02.591+0000 7f003150d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:14:02.933 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:02.685+0000 7f003150d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:14:02.933 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:02.829+0000 7f003150d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:14:03.246 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:02.930+0000 7f003150d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:14:03.246 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.019+0000 7f003150d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:14:03.503 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.298+0000 7f003150d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:14:03.503 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.414+0000 7f003150d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:03.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-mon[25694]: mgrmap e30: y(active, since 1.33365s) 2024-02-13T15:14:03.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:02] ENGINE Bus STARTING 2024-02-13T15:14:03.503 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:02] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:03.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:02] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:03.504 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:02] ENGINE Bus STARTED 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[24754]: mgrmap e30: y(active, since 1.33365s) 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[24754]: [13/Feb/2024:15:14:02] ENGINE Bus STARTING 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[24754]: [13/Feb/2024:15:14:02] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[24754]: [13/Feb/2024:15:14:02] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[24754]: [13/Feb/2024:15:14:02] ENGINE Bus STARTED 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[29232]: mgrmap e30: y(active, since 1.33365s) 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:02] ENGINE Bus STARTING 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:02] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:03.510 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:02] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:03.511 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:02] ENGINE Bus STARTED 2024-02-13T15:14:03.798 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.501+0000 7f003150d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:14:03.798 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.602+0000 7f003150d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:14:03.798 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:03.796+0000 7f003150d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:14:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:14:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:14:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:14:04.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:04 smithi002 ceph-mon[24754]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:04.288 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:04 smithi002 ceph-mon[24754]: mgrmap e31: y(active, since 2s) 2024-02-13T15:14:04.289 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:04 smithi002 ceph-mon[29232]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:04.289 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:04 smithi002 ceph-mon[29232]: mgrmap e31: y(active, since 2s) 2024-02-13T15:14:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:04 smithi098 ceph-mon[25694]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:04 smithi098 ceph-mon[25694]: mgrmap e31: y(active, since 2s) 2024-02-13T15:14:05.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:04.797+0000 7f003150d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:14:05.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:04.930+0000 7f003150d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:14:05.580 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:05.279+0000 7f003150d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:14:05.580 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:05.370+0000 7f003150d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:14:05.580 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:05.578+0000 7f003150d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:05.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:05.771+0000 7f003150d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:14:06.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:06.075+0000 7f003150d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:06.601 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:06.346+0000 7f003150d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:06.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:06.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: mgrmap e32: y(active, since 4s) 2024-02-13T15:14:06.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.602 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.602 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: mgrmap e32: y(active, since 4s) 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:06.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: mgrmap e32: y(active, since 4s) 2024-02-13T15:14:06.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:06.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:06 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:07.081 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:06.599+0000 7f003150d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:14:07.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:07.446+0000 7f003150d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:14:08.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:07 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.021 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:07 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.021 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:07 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.021 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:07 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:07 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:08.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:08.018+0000 7f003150d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:14:08.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:08.118+0000 7f003150d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:14:08.715 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:08.341+0000 7f003150d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:14:08.715 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:14:08] ENGINE Bus STARTING 2024-02-13T15:14:08.715 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:14:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T15:14:08.716 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: CherryPy Checker: 2024-02-13T15:14:08.716 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: The Application mounted at '' has an empty config. 2024-02-13T15:14:08.716 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: 2024-02-13T15:14:08.716 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:14:08] ENGINE Serving on http://:::9283 2024-02-13T15:14:08.716 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[46900]: [13/Feb/2024:15:14:08] ENGINE Bus STARTED 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:08 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: Standby manager daemon x started 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:08 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/264042852' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:10 smithi098 ceph-mon[25694]: mgrmap e33: y(active, since 8s), standbys: x 2024-02-13T15:14:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:10 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:10.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:10 smithi098 ceph-mon[25694]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:14:10.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[24754]: mgrmap e33: y(active, since 8s), standbys: x 2024-02-13T15:14:10.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:10.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[24754]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:14:10.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[29232]: mgrmap e33: y(active, since 8s), standbys: x 2024-02-13T15:14:10.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:10.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:10 smithi002 ceph-mon[29232]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:14:11.382 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:14:11.423 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.423 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.423 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.425 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:11 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:11.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:11 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.128 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:14:12.128 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (67s) 6s ago 6m 17.1M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (31s) 6s ago 5m 45.2M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 unknown 6s ago 4m - - 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283 running (112s) 6s ago 10m 74.8M - 19.0.0-1423-ge1b79d5a 28480620e5ed ce1243ea34d0 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (25s) 6s ago 12m 473M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (12m) 6s ago 12m 55.3M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (10m) 6s ago 10m 47.1M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (10m) 6s ago 10m 45.1M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (77s) 6s ago 6m 9995k - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (46s) 6s ago 6m 11.7M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (10m) 6s ago 10m 52.0M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (9m) 6s ago 9m 53.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (9m) 6s ago 9m 48.2M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:14:12.129 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (8m) 6s ago 8m 53.3M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (8m) 6s ago 8m 52.4M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (8m) 6s ago 8m 50.2M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (7m) 6s ago 7m 49.0M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (7m) 6s ago 7m 50.8M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (48s) 6s ago 6m 54.9M - 2.33.4 514e6a882f6e f8c6aaae9c93 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (5m) 6s ago 5m 72.5M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (5m) 6s ago 5m 73.5M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (5m) 6s ago 5m 71.0M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:14:12.130 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (4m) 6s ago 4m 73.4M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:14:12.544 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:14:12.545 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:14:12.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[24754]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.546 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.547 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:14:12.547 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:14:12.547 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.547 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:12 smithi002 ceph-mon[29232]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: Reconfiguring daemon prometheus.a on smithi098 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:12.564 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:12 smithi098 ceph-mon[25694]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:14:12.889 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:14:12.889 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:14:12.889 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:14:12.889 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "2/23 daemons upgraded", 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:14:12.890 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:14:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-13T15:14:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-13T15:14:13.292 INFO:teuthology.orchestra.run.smithi002.stdout: daemon iscsi.foo.smithi002.fozvhf on smithi002 is in unknown state 2024-02-13T15:14:13.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:13 smithi098 ceph-mon[25694]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:13.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:13 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/396890496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:13.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:13 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1743815823' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:14:13.521 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:13 smithi098 ceph-mon[25694]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:13.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:14:13.196Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=7 2024-02-13T15:14:13.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[24754]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:13.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/396890496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:13.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/1743815823' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:14:13.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[24754]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:13.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[29232]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:13.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/396890496' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:13.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1743815823' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:14:13.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:13 smithi002 ceph-mon[29232]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:14 smithi098 ceph-mon[25694]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:14 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/222360726' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:14.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:14 smithi002 ceph-mon[24754]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:14.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:14 smithi002 ceph-mon[24754]: from='client.? 172.21.15.2:0/222360726' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:14 smithi002 ceph-mon[29232]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:14.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:14 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/222360726' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:15.217 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 systemd[1]: Stopping Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:15.217 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 bash[53804]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.216Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T15:14:15.502 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.217Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T15:14:15.503 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.217Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-13T15:14:15.503 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[50511]: ts=2024-02-13T15:14:15.217Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-13T15:14:15.503 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 podman[53817]: 2024-02-13 15:14:15.226545665 +0000 UTC m=+0.123362550 container died f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:15.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 podman[53817]: 2024-02-13 15:14:15.591019178 +0000 UTC m=+0.487836063 container cleanup f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:15.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 bash[53817]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T15:14:15.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 bash[53843]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:14:16.119 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 podman[53831]: 2024-02-13 15:14:15.867811398 +0000 UTC m=+0.642441251 container remove f8c6aaae9c939370d9d485ec6ff95c55dc61aed98f75b3ea921a14232e0aac1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:16.120 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-13T15:14:16.120 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 systemd[1]: Stopped Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:16.120 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:15 smithi098 systemd[1]: Starting Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:16.120 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 podman[53915]: 2024-02-13 15:14:16.011646408 +0000 UTC m=+0.018812446 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-13T15:14:16.120 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 podman[53915]: 2024-02-13 15:14:16.11882823 +0000 UTC m=+0.125994312 container create 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:16.372 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 podman[53915]: 2024-02-13 15:14:16.346140609 +0000 UTC m=+0.353306648 container init 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:16.372 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 podman[53915]: 2024-02-13 15:14:16.348756421 +0000 UTC m=+0.355922461 container start 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:14:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-mon[25694]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.371Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.371Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.371Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.371Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi098 (none))" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.372Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.372Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.373Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.374Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.375Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.377Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.377Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.93µs 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.377Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-13T15:14:16.833 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 bash[53915]: 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 2024-02-13T15:14:16.834 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 systemd[1]: Started Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:16.834 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.579Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-02-13T15:14:16.834 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.831Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-02-13T15:14:16.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[24754]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:16.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:16.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:16.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[29232]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:14:16.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:16.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:16 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.878Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.878Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.878Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.912µs wal_replay_duration=501.187671ms total_replay_duration=501.23316ms 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.882Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.882Z caller=main.go:947 level=info msg="TSDB started" 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.882Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.903Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.77475ms db_storage=659ns remote_storage=1.348µs web_handler=306ns query_engine=630ns scrape=399.37µs scrape_sd=82.109µs notify=11.194µs notify_sd=10.027µs rules=20.025203ms 2024-02-13T15:14:17.099 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:16.903Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-13T15:14:17.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:17 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:17 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:17 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:17 smithi098 ceph-mon[25694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.747 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:17 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[24754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[29232]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T15:14:17.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:17 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:18.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:18 smithi098 ceph-mon[25694]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:18.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:18 smithi098 ceph-mon[25694]: Upgrade: Updating mgr.x 2024-02-13T15:14:18.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:18 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:14:18.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:18 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:14:18.645 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:18 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[24754]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[24754]: Upgrade: Updating mgr.x 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:18.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[29232]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T15:14:18.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[29232]: Upgrade: Updating mgr.x 2024-02-13T15:14:18.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:14:18.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:14:18.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:18 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:19.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:19 smithi002 ceph-mon[24754]: Deploying daemon mgr.x on smithi098 2024-02-13T15:14:19.778 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:19 smithi002 ceph-mon[29232]: Deploying daemon mgr.x on smithi098 2024-02-13T15:14:19.796 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:19 smithi098 ceph-mon[25694]: Deploying daemon mgr.x on smithi098 2024-02-13T15:14:20.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:20 smithi098 systemd[1]: Stopping Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:20.644 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:20 smithi098 ceph-mon[25694]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:20.644 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:20 smithi098 podman[54270]: 2024-02-13 15:14:20.380299729 +0000 UTC m=+0.165362006 container died ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True) 2024-02-13T15:14:20.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:20 smithi002 ceph-mon[24754]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:20.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:20 smithi002 ceph-mon[29232]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:21.022 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:20 smithi098 podman[54270]: 2024-02-13 15:14:20.749623032 +0000 UTC m=+0.534685302 container cleanup ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-13T15:14:21.022 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:20 smithi098 bash[54270]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x 2024-02-13T15:14:21.331 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 podman[54284]: 2024-02-13 15:14:21.021281175 +0000 UTC m=+0.638416568 container remove ce1243ea34d010ed019393d310845cc027f0016b7a81e244886ba76ed7680c1e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-13T15:14:21.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:14:21.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-13T15:14:21.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: Stopped Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:21.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service: Consumed 28.197s CPU time. 2024-02-13T15:14:21.628 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: Starting Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:21.628 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 podman[54414]: 2024-02-13 15:14:21.482170841 +0000 UTC m=+0.018466392 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:14:21.628 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 podman[54414]: 2024-02-13 15:14:21.627419199 +0000 UTC m=+0.163714749 container create 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD) 2024-02-13T15:14:21.914 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 podman[54414]: 2024-02-13 15:14:21.829823018 +0000 UTC m=+0.366118570 container init 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:14:21.914 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 podman[54414]: 2024-02-13 15:14:21.832462539 +0000 UTC m=+0.368758092 container start 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-13T15:14:21.914 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 ceph-mgr[54434]: -- 172.21.15.98:0/1608216612 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5617c261a1a0 con 0x5617c25f0800 2024-02-13T15:14:22.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 bash[54414]: 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 2024-02-13T15:14:22.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:21 smithi098 systemd[1]: Started Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:22.810 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:22.404+0000 7f08f957d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:14:22.810 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:22.518+0000 7f08f957d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:14:23.069 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:22 smithi098 ceph-mon[25694]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:23.069 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:22 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.069 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:22 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.189+0000 7f08f957d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:14:23.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.285+0000 7f08f957d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:14:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[29232]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[24754]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 509 B/s rd, 0 op/s 2024-02-13T15:14:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:22 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:23.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.437+0000 7f08f957d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:14:23.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.534+0000 7f08f957d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:14:23.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.620+0000 7f08f957d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:14:24.225 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-mon[25694]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:24.226 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:23 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:23.896+0000 7f08f957d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:14:24.226 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:24.014+0000 7f08f957d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:24.226 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:24.113+0000 7f08f957d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:14:24.226 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:24.224+0000 7f08f957d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:14:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:23 smithi002 ceph-mon[24754]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:24.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:23 smithi002 ceph-mon[29232]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:24.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:24 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:24.403+0000 7f08f957d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:14:25.327 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:24 smithi098 ceph-mon[25694]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:24 smithi002 ceph-mon[24754]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:25.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:24 smithi002 ceph-mon[29232]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:25.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:25.436+0000 7f08f957d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:14:25.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:25.568+0000 7f08f957d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:14:26.240 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:25 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:25.923+0000 7f08f957d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:14:26.240 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:26.031+0000 7f08f957d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:14:26.240 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:26.238+0000 7f08f957d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:26.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:26.437+0000 7f08f957d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:14:26.995 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:26.742+0000 7f08f957d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:27.268 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:27.013+0000 7f08f957d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:27.268 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:27.266+0000 7f08f957d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:14:27.268 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-mon[25694]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:14:27.268 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:27.268 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:26 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[24754]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[29232]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:27.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:26 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.420 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:28.165+0000 7f08f957d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:14:28.762 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:28.761+0000 7f08f957d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:14:28.763 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-mon[25694]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:14:28.763 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.763 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[29232]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[24754]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:28.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:28 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:29.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:28.863+0000 7f08f957d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:14:29.451 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:29.091+0000 7f08f957d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:14:29.451 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:29] ENGINE Bus STARTING 2024-02-13T15:14:29.452 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: CherryPy Checker: 2024-02-13T15:14:29.452 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: The Application mounted at '' has an empty config. 2024-02-13T15:14:29.452 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:29.452 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:29] ENGINE Serving on http://:::9283 2024-02-13T15:14:29.452 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:29] ENGINE Bus STARTED 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: Standby manager daemon x restarted 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:29.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:29 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: Standby manager daemon x restarted 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: Standby manager daemon x started 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[24754]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: Standby manager daemon x restarted 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:29.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/1009454662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:30.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:14:29] "GET /metrics HTTP/1.1" 200 38217 "" "Prometheus/2.33.4" 2024-02-13T15:14:30.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:30 smithi002 ceph-mon[24754]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:30.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:30 smithi002 ceph-mon[24754]: mgrmap e34: y(active, since 28s), standbys: x 2024-02-13T15:14:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:30 smithi002 ceph-mon[29232]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:30 smithi002 ceph-mon[29232]: mgrmap e34: y(active, since 28s), standbys: x 2024-02-13T15:14:31.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:30 smithi098 ceph-mon[25694]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:14:31.002 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:30 smithi098 ceph-mon[25694]: mgrmap e34: y(active, since 28s), standbys: x 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:32.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-13T15:14:32.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:32 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-13T15:14:32.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:32 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: Upgrade: Setting container_image for all mgr 2024-02-13T15:14:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: Upgrade: It appears safe to stop mon.a 2024-02-13T15:14:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[24754]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: Upgrade: Setting container_image for all mgr 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: Upgrade: It appears safe to stop mon.a 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:33 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: Upgrade: Setting container_image for all mgr 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: Upgrade: It appears safe to stop mon.a 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:33 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[24754]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[24754]: Upgrade: Updating mon.a 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[24754]: Deploying daemon mon.a on smithi002 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[24754]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[29232]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:34.249 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[29232]: Upgrade: Updating mon.a 2024-02-13T15:14:34.250 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[29232]: Deploying daemon mon.a on smithi002 2024-02-13T15:14:34.250 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:34 smithi002 ceph-mon[29232]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:34 smithi098 ceph-mon[25694]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:34 smithi098 ceph-mon[25694]: Upgrade: Updating mon.a 2024-02-13T15:14:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:34 smithi098 ceph-mon[25694]: Deploying daemon mon.a on smithi002 2024-02-13T15:14:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:34 smithi098 ceph-mon[25694]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:35.096 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:34 smithi002 systemd[1]: Stopping Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[24750]: 2024-02-13T15:14:35.093+0000 7fb72be10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[24750]: 2024-02-13T15:14:35.093+0000 7fb72be10700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 podman[64292]: 2024-02-13 15:14:35.125788136 +0000 UTC m=+0.155008754 container died 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, ceph=True) 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE Bus STOPPING 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE Bus STOPPED 2024-02-13T15:14:35.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE Bus STARTING 2024-02-13T15:14:35.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE Serving on http://:::9283 2024-02-13T15:14:35.377 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:35 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:35] ENGINE Bus STARTED 2024-02-13T15:14:35.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 podman[64292]: 2024-02-13 15:14:35.494977102 +0000 UTC m=+0.524197762 container cleanup 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, io.openshift.expose-services=, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_CLEAN=True) 2024-02-13T15:14:35.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 bash[64292]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a 2024-02-13T15:14:36.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 podman[64317]: 2024-02-13 15:14:35.79620308 +0000 UTC m=+0.665619083 container remove 3c4d816ebc8a8fd5958a4db873d0b29a1db655edd6cc192706cdbdce4f85c82d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-02-13T15:14:36.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-13T15:14:36.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 systemd[1]: Stopped Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:36.124 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:35 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service: Consumed 14.068s CPU time. 2024-02-13T15:14:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 systemd[1]: Starting Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:14:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 podman[64450]: 2024-02-13 15:14:36.239154872 +0000 UTC m=+0.017731673 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:14:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 podman[64450]: 2024-02-13 15:14:36.355466624 +0000 UTC m=+0.134043415 container create 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2) 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 podman[64450]: 2024-02-13 15:14:36.59214615 +0000 UTC m=+0.370722949 container init 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 podman[64450]: 2024-02-13 15:14:36.595286483 +0000 UTC m=+0.373863291 container start 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 2 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: pidfile_write: ignore empty --pid-file 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: load: jerasure load: lrc 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: RocksDB version: 7.9.2 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Git sha 0 2024-02-13T15:14:36.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: DB SUMMARY 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: DB Session ID: EMIDPFS4IA9Z54X3O63N 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: CURRENT file: CURRENT 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3588507 ; 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.error_if_exists: 0 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.create_if_missing: 0 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.paranoid_checks: 1 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.env: 0x55d2195b9ae0 2024-02-13T15:14:36.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.info_log: 0x55d21bcaf8c0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.statistics: (nil) 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.use_fsync: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_log_file_size: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_fallocate: 1 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.use_direct_reads: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T15:14:36.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.db_log_dir: 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.wal_dir: 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.write_buffer_manager: 0x55d21bc83400 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T15:14:36.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.unordered_write: 0 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.row_cache: None 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.wal_filter: None 2024-02-13T15:14:36.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.two_write_queues: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.wal_compression: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.atomic_flush: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T15:14:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.log_readahead_size: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_background_jobs: 2 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_background_compactions: -1 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_subcompactions: 1 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T15:14:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_open_files: -1 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_background_flushes: -1 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Compression algorithms supported: 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kZSTD supported: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kXpressCompression supported: 0 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T15:14:36.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kZlibCompression supported: 1 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kSnappyCompression supported: 1 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kLZ4Compression supported: 1 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: kBZip2Compression supported: 0 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.merge_operator: 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_filter: None 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d21bcaf9c0) 2024-02-13T15:14:36.885 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55d21bcb3090 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-02-13T15:14:36.886 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-02-13T15:14:36.887 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression: NoCompression 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.num_levels: 7 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T15:14:36.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:14:36.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T15:14:36.890 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.table_properties_collectors: 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.inplace_update_support: 0 2024-02-13T15:14:36.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.bloom_locality: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.max_successive_merges: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.ttl: 2592000 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enable_blob_files: false 2024-02-13T15:14:36.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.min_blob_size: 0 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12099, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e10bc867-dd77-4abd-8e5f-be9af9b1ccdd 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837276635468, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837276645413, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3294538, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12100, "largest_seqno": 13637, "table_properties": {"data_size": 3287155, "index_size": 4636, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1733, "raw_key_size": 15822, "raw_average_key_size": 23, "raw_value_size": 3272437, "raw_average_value_size": 4891, "num_data_blocks": 212, "num_entries": 669, "num_filter_entries": 669, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707837276, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e10bc867-dd77-4abd-8e5f-be9af9b1ccdd", "db_session_id": "EMIDPFS4IA9Z54X3O63N", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-13T15:14:36.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837276645520, "job": 1, "event": "recovery_finished"} 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d21bda4000 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: DB pointer 0x55d21bd8e000 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 1/0 3.14 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 630.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: L6 1/0 7.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 10.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 630.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 630.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 630.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:14:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 137.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 137.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x55d21bcb3090#2 capacity: 512.00 MB usage: 6.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???) e3 preinit fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).mds e1 new map 2024-02-13T15:14:36.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).mds e1 print_map 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).osd e103 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 ceph-mon[64469]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 bash[64450]: 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 2024-02-13T15:14:36.897 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:36 smithi002 systemd[1]: Started Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:14:39.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: mon.a calling monitor election 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: fsmap 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: mgrmap e34: y(active, since 38s), standbys: x 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: daemon iscsi.foo.smithi002.fozvhf on smithi002 is in unknown state 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: from='mgr.25195 ' entity='' 2024-02-13T15:14:40.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[64469]: mgrmap e35: y(active, since 38s), standbys: x 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ignoring --setuser ceph since I am not root 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ignoring --setgroup ceph since I am not root 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mgr[60181]: -- 172.21.15.2:0/901245939 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564480cae1a0 con 0x564480c84800 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: mon.a calling monitor election 2024-02-13T15:14:40.073 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: fsmap 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: mgrmap e34: y(active, since 38s), standbys: x 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: daemon iscsi.foo.smithi002.fozvhf on smithi002 is in unknown state 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: from='mgr.25195 ' entity='' 2024-02-13T15:14:40.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:39 smithi002 ceph-mon[29232]: mgrmap e35: y(active, since 38s), standbys: x 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: ignoring --setuser ceph since I am not root 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: ignoring --setgroup ceph since I am not root 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mgr[54434]: -- 172.21.15.98:0/2095028137 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a0bd741a0 con 0x555a0bd4b000 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: from='mgr.25195 172.21.15.2:0/1763208084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:14:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: mon.a calling monitor election 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: fsmap 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: osdmap e103: 8 total, 8 up, 8 in 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: mgrmap e34: y(active, since 38s), standbys: x 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: daemon iscsi.foo.smithi002.fozvhf on smithi002 is in unknown state 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: from='mgr.25195 ' entity='' 2024-02-13T15:14:40.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:39 smithi098 ceph-mon[25694]: mgrmap e35: y(active, since 38s), standbys: x 2024-02-13T15:14:40.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:40.280+0000 7fe25730f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:14:40.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:40 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:40.258+0000 7f172887b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T15:14:40.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:40 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:40.351+0000 7f172887b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:14:40.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:40.376+0000 7fe25730f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T15:14:41.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.003+0000 7f172887b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:14:41.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.098+0000 7f172887b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:14:41.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.241+0000 7f172887b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:14:41.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.037+0000 7fe25730f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T15:14:41.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.133+0000 7fe25730f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T15:14:41.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.277+0000 7fe25730f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T15:14:41.689 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.375+0000 7fe25730f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:14:41.689 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.461+0000 7fe25730f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:14:41.690 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.338+0000 7f172887b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T15:14:41.690 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.422+0000 7f172887b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T15:14:41.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-mon[25694]: mgrmap e36: y(active, since 39s), standbys: x 2024-02-13T15:14:41.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.692+0000 7f172887b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:14:41.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.804+0000 7f172887b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:41.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.889+0000 7f172887b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:14:41.984 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:41.983+0000 7f172887b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:14:42.035 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-mon[64469]: mgrmap e36: y(active, since 39s), standbys: x 2024-02-13T15:14:42.035 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-mon[29232]: mgrmap e36: y(active, since 39s), standbys: x 2024-02-13T15:14:42.035 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.732+0000 7fe25730f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T15:14:42.035 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.846+0000 7fe25730f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:42.035 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:41.935+0000 7fe25730f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T15:14:42.331 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:42.160+0000 7f172887b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:14:42.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:42.033+0000 7fe25730f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T15:14:42.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:42.212+0000 7fe25730f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T15:14:43.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:43.119+0000 7f172887b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:14:43.582 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:43.247+0000 7f172887b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:14:43.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:43.207+0000 7fe25730f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T15:14:43.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:43.344+0000 7fe25730f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T15:14:43.874 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:43.582+0000 7f172887b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:14:43.874 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:43.674+0000 7f172887b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:14:43.874 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:43.872+0000 7f172887b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:43.994 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:43.693+0000 7fe25730f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T15:14:43.994 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:43.787+0000 7fe25730f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T15:14:43.994 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:43 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:43.992+0000 7fe25730f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:44.332 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:44.058+0000 7f172887b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:14:44.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:44.186+0000 7fe25730f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T15:14:44.610 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:44.347+0000 7f172887b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:44.611 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:44.608+0000 7f172887b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:44.758 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:44.490+0000 7fe25730f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T15:14:45.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:44.850+0000 7f172887b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:14:45.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:44.757+0000 7fe25730f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T15:14:45.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:45 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:45.005+0000 7fe25730f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T15:14:46.082 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:45.655+0000 7f172887b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:14:46.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:45 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:45.852+0000 7fe25730f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T15:14:46.550 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:46.228+0000 7f172887b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:14:46.550 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:46.327+0000 7f172887b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:46.439+0000 7fe25730f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:46.543+0000 7fe25730f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: Standby manager daemon x restarted 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: Standby manager daemon x started 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:46.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[64469]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: Standby manager daemon x restarted 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: Standby manager daemon x started 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:46.779 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-mon[29232]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: Standby manager daemon x restarted 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: Standby manager daemon x started 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-mon[25694]: from='mgr.? 172.21.15.98:0/4196901534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T15:14:46.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b[25677]: 2024-02-13T15:14:46.793+0000 7f1729f83700 -1 mon.b@2(peon).osd e104 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:46.548+0000 7f172887b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:46] ENGINE Bus STARTING 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: CherryPy Checker: 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: The Application mounted at '' has an empty config. 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:46] ENGINE Serving on http://:::9283 2024-02-13T15:14:46.833 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:14:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x[54430]: [13/Feb/2024:15:14:46] ENGINE Bus STARTED 2024-02-13T15:14:47.033 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:46.776+0000 7fe25730f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T15:14:47.033 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:46] ENGINE Bus STARTING 2024-02-13T15:14:47.033 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c[29228]: 2024-02-13T15:14:46.793+0000 7f45f5365700 -1 mon.c@1(peon).osd e104 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-13T15:14:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: CherryPy Checker: 2024-02-13T15:14:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: The Application mounted at '' has an empty config. 2024-02-13T15:14:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:47] ENGINE Serving on http://:::9283 2024-02-13T15:14:47.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:14:47] ENGINE Bus STARTED 2024-02-13T15:14:47.832 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:14:47.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:14:47.801+0000 7fe20d50d700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: mgrmap e37: y(active, since 45s), standbys: x 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: Active manager daemon y restarted 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: Activating manager daemon y 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: mgrmap e38: y(active, starting, since 0.00507214s), standbys: x 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:47.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: Manager daemon y is now available 2024-02-13T15:14:47.878 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: mgrmap e37: y(active, since 45s), standbys: x 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: Active manager daemon y restarted 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: Activating manager daemon y 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: mgrmap e38: y(active, starting, since 0.00507214s), standbys: x 2024-02-13T15:14:47.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:47.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: Manager daemon y is now available 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:47.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:47 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: mgrmap e37: y(active, since 45s), standbys: x 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: Active manager daemon y restarted 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: Activating manager daemon y 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: mgrmap e38: y(active, starting, since 0.00507214s), standbys: x 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:14:48.025 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: Manager daemon y is now available 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:48.026 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:47 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T15:14:48.520 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:14:48.520 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (104s) 42s ago 6m 17.1M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:14:48.520 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (67s) 22s ago 6m 45.3M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 unknown 42s ago 5m - - 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (27s) 22s ago 11m 67.0M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (61s) 42s ago 12m 473M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (12m) 42s ago 12m 55.3M 2048M 17.2.0 e1d6a67b021e 3c4d816ebc8a 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (11m) 22s ago 11m 48.9M 2048M 17.2.0 e1d6a67b021e 42fd2168af81 2024-02-13T15:14:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (11m) 42s ago 11m 45.1M 2048M 17.2.0 e1d6a67b021e c2812dc14cb1 2024-02-13T15:14:48.522 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (113s) 42s ago 7m 9995k - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:14:48.522 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (82s) 22s ago 7m 19.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:14:48.522 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (10m) 42s ago 10m 52.0M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:14:48.522 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (10m) 42s ago 10m 53.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (9m) 42s ago 9m 48.2M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (9m) 42s ago 9m 53.3M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (9m) 22s ago 9m 52.9M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (8m) 22s ago 8m 50.6M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (8m) 22s ago 8m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (8m) 22s ago 8m 51.4M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (32s) 22s ago 7m 44.4M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (5m) 42s ago 5m 72.5M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (5m) 22s ago 5m 73.9M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (5m) 42s ago 5m 71.0M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:14:48.523 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (5m) 22s ago 5m 73.7M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:14:48.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[64469]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[64469]: mgrmap e39: y(active, since 1.0119s), standbys: x 2024-02-13T15:14:48.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[29232]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:48 smithi002 ceph-mon[29232]: mgrmap e39: y(active, since 1.0119s), standbys: x 2024-02-13T15:14:48.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:48 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:48 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:48 smithi098 ceph-mon[25694]: failed to encode map e104 with expected crc 2024-02-13T15:14:48.883 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:48 smithi098 ceph-mon[25694]: mgrmap e39: y(active, since 1.0119s), standbys: x 2024-02-13T15:14:48.948 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:14:48.948 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:14:48.948 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-13T15:14:48.948 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:14:48.949 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:14:49.304 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:14:49.304 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:14:49.304 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:14:49.304 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "2/23 daemons upgraded", 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:14:49.305 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:14:49.738 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:14:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: [13/Feb/2024:15:14:48] ENGINE Bus STARTING 2024-02-13T15:14:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: [13/Feb/2024:15:14:48] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: [13/Feb/2024:15:14:48] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: [13/Feb/2024:15:14:48] ENGINE Bus STARTED 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1114233736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:48] ENGINE Bus STARTING 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:48] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:48] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: [13/Feb/2024:15:14:48] ENGINE Bus STARTED 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: Cluster is now healthy 2024-02-13T15:14:49.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/1114233736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:48] ENGINE Bus STARTING 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:48] ENGINE Serving on https://172.21.15.2:7150 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:48] ENGINE Serving on http://172.21.15.2:8765 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: [13/Feb/2024:15:14:48] ENGINE Bus STARTED 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: Cluster is now healthy 2024-02-13T15:14:49.905 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:49 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/1114233736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:50.204 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:14:49] "GET /metrics HTTP/1.1" 200 35425 "" "Prometheus/2.33.4" 2024-02-13T15:14:51.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[64469]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[64469]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3269590305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[64469]: mgrmap e40: y(active, since 3s), standbys: x 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[29232]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[29232]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:51.006 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[29232]: from='client.? 172.21.15.2:0/3269590305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:51.007 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:50 smithi002 ceph-mon[29232]: mgrmap e40: y(active, since 3s), standbys: x 2024-02-13T15:14:51.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:50 smithi098 ceph-mon[25694]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:51.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:50 smithi098 ceph-mon[25694]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:14:51.019 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:50 smithi098 ceph-mon[25694]: from='client.? 172.21.15.2:0/3269590305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:14:51.020 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:50 smithi098 ceph-mon[25694]: mgrmap e40: y(active, since 3s), standbys: x 2024-02-13T15:14:52.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: mgrmap e41: y(active, since 4s), standbys: x 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: mgrmap e41: y(active, since 4s), standbys: x 2024-02-13T15:14:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:52.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:52 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: mgrmap e41: y(active, since 4s), standbys: x 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:52 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:53 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:53 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:53 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:53 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:53 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:54 smithi098 ceph-mon[25694]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:55.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:54 smithi002 ceph-mon[64469]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:55.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:54 smithi002 ceph-mon[29232]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:14:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:56 smithi098 ceph-mon[25694]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:14:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:56 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:56 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:56 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:56 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[64469]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:14:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[29232]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:57.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:56 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.927 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:57.928 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:57.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-13T15:14:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-13T15:14:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.conf 2024-02-13T15:14:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T15:14:58.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: Updating smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:14:58.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:57 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-13T15:14:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: Upgrade: It appears safe to stop mon.c 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:14:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: Upgrade: It appears safe to stop mon.c 2024-02-13T15:14:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:58 smithi002 ceph-mon[29232]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:58.983 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:14:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: Updating smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: Upgrade: It appears safe to stop mon.c 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:14:58.984 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:58 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:14:59.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:59.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[64469]: Upgrade: Updating mon.c 2024-02-13T15:14:59.857 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[64469]: Deploying daemon mon.c on smithi002 2024-02-13T15:14:59.857 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[29232]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:59.857 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[29232]: Upgrade: Updating mon.c 2024-02-13T15:14:59.857 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-mon[29232]: Deploying daemon mon.c on smithi002 2024-02-13T15:14:59.993 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:59 smithi098 ceph-mon[25694]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:14:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:59 smithi098 ceph-mon[25694]: Upgrade: Updating mon.c 2024-02-13T15:14:59.994 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:14:59 smithi098 ceph-mon[25694]: Deploying daemon mon.c on smithi002 2024-02-13T15:15:00.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:14:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:14:59] "GET /metrics HTTP/1.1" 200 35425 "" "Prometheus/2.33.4" 2024-02-13T15:15:00.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-mon[64469]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:15:00.826 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:00 smithi002 systemd[1]: Stopping Ceph mon.c for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:15:00.826 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-mon[29232]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:15:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:00 smithi098 ceph-mon[25694]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-13T15:15:01.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c[29228]: 2024-02-13T15:15:00.823+0000 7f45fab70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:15:01.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c[29228]: 2024-02-13T15:15:00.823+0000 7f45fab70700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-13T15:15:01.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:00 smithi002 podman[66849]: 2024-02-13 15:15:00.854258981 +0000 UTC m=+0.126700492 container died c2812dc14cb19d1843bb90f1e369fe9ad00825e52143b6a74d5567eaa8c193cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, name=centos-stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64) 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE Bus STOPPING 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE Bus STOPPED 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE Bus STARTING 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE Serving on http://:::9283 2024-02-13T15:15:01.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:00] ENGINE Bus STARTED 2024-02-13T15:15:01.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 podman[66849]: 2024-02-13 15:15:01.265235508 +0000 UTC m=+0.537677018 container cleanup c2812dc14cb19d1843bb90f1e369fe9ad00825e52143b6a74d5567eaa8c193cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, architecture=x86_64, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, vendor=Red Hat, Inc., RELEASE=HEAD, version=8, release=754) 2024-02-13T15:15:01.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 bash[66849]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c 2024-02-13T15:15:01.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 podman[66865]: 2024-02-13 15:15:01.483043822 +0000 UTC m=+0.630847167 container remove c2812dc14cb19d1843bb90f1e369fe9ad00825e52143b6a74d5567eaa8c193cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-13T15:15:01.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-13T15:15:01.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 systemd[1]: Stopped Ceph mon.c for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:15:01.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.c.service: Consumed 8.056s CPU time. 2024-02-13T15:15:02.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:01 smithi002 systemd[1]: Starting Ceph mon.c for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:15:02.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 podman[67005]: 2024-02-13 15:15:02.018012136 +0000 UTC m=+0.017686210 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:15:02.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 podman[67005]: 2024-02-13 15:15:02.125595202 +0000 UTC m=+0.125269315 container create 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 podman[67005]: 2024-02-13 15:15:02.369647649 +0000 UTC m=+0.369321727 container init 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 podman[67005]: 2024-02-13 15:15:02.372780963 +0000 UTC m=+0.372455046 container start 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 2 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: pidfile_write: ignore empty --pid-file 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: load: jerasure load: lrc 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: RocksDB version: 7.9.2 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Git sha 0 2024-02-13T15:15:02.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: DB SUMMARY 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: DB Session ID: AFL5YULXCXHDV5PU4D6R 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: CURRENT file: CURRENT 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 281035 ; 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.error_if_exists: 0 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.create_if_missing: 0 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.paranoid_checks: 1 2024-02-13T15:15:02.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.env: 0x556425e27ae0 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.info_log: 0x5564277ed8c0 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T15:15:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.statistics: (nil) 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.use_fsync: 0 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_log_file_size: 0 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_fallocate: 1 2024-02-13T15:15:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.use_direct_reads: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.db_log_dir: 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.wal_dir: 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T15:15:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.write_buffer_manager: 0x5564277c1400 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.unordered_write: 0 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T15:15:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.row_cache: None 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.wal_filter: None 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.two_write_queues: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.wal_compression: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.atomic_flush: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.log_readahead_size: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T15:15:02.633 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_background_jobs: 2 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_background_compactions: -1 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_subcompactions: 1 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T15:15:02.634 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_open_files: -1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_background_flushes: -1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Compression algorithms supported: 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kZSTD supported: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kXpressCompression supported: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kZlibCompression supported: 1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kSnappyCompression supported: 1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kLZ4Compression supported: 1 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: kBZip2Compression supported: 0 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T15:15:02.635 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.merge_operator: 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_filter: None 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5564277ed9c0) 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_type: 0 2024-02-13T15:15:02.636 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_index_type: 0 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_shortening: 1 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: checksum: 4 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: no_block_cache: 0 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache: 0x5564277f1090 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_options: 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: capacity : 536870912 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: num_shard_bits : 4 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: strict_capacity_limit : 0 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-02-13T15:15:02.637 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_compressed: (nil) 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: persistent_cache: (nil) 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size: 4096 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size_deviation: 10 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_restart_interval: 16 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_block_restart_interval: 1 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: metadata_block_size: 4096 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: partition_filters: 0 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: use_delta_encoding: 1 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: filter_policy: bloomfilter 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: whole_key_filtering: 1 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: verify_compression: 0 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: format_version: 5 2024-02-13T15:15:02.638 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_index_compression: 1 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_align: 0 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout: max_auto_readahead_size: 262144 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout: prepopulate_block_cache: 0 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout: initial_auto_readahead_size: 8192 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression: NoCompression 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.num_levels: 7 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T15:15:02.639 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T15:15:02.640 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T15:15:02.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T15:15:02.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T15:15:02.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T15:15:02.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T15:15:02.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T15:15:02.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T15:15:02.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.table_properties_collectors: 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.inplace_update_support: 0 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.bloom_locality: 0 2024-02-13T15:15:02.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.max_successive_merges: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.ttl: 2592000 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enable_blob_files: false 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.min_blob_size: 0 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T15:15:02.645 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15173, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d3aab995-0c34-4ef5-91c8-bec09082d947 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837302401634, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837302403132, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 187419, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15178, "largest_seqno": 15296, "table_properties": {"data_size": 185784, "index_size": 431, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1822, "raw_average_key_size": 26, "raw_value_size": 184148, "raw_average_value_size": 2708, "num_data_blocks": 17, "num_entries": 68, "num_filter_entries": 68, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707837302, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d3aab995-0c34-4ef5-91c8-bec09082d947", "db_session_id": "AFL5YULXCXHDV5PU4D6R", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837302403218, "job": 1, "event": "recovery_finished"} 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-13T15:15:02.646 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5564278e2000 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: DB pointer 0x5564278cc000 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** DB Stats ** 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T15:15:02.647 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: L0 1/0 183.03 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 181.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: L6 1/0 11.53 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Sum 2/0 11.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 181.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 181.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 181.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.648 INFO:journalctl@ceph.mon.c.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative compaction: 0.00 GB write, 50.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval compaction: 0.00 GB write, 50.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Block cache BinnedLRUCache@0x5564277f1090#2 capacity: 512.00 MB usage: 0.75 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-13T15:15:02.649 INFO:journalctl@ceph.mon.c.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.52 KB,9.83477e-05%) Misc(1,0.00 KB,0%) 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: starting mon.c rank 1 at public addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] at bind addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???) e3 preinit fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).mds e1 new map 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).mds e1 print_map 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: e1 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: legacy client fscid: -1 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout: No filesystems configured 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:02.650 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:02.651 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:02.651 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 ceph-mon[67023]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-02-13T15:15:02.651 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 bash[67005]: 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 2024-02-13T15:15:02.651 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:02 smithi002 systemd[1]: Started Ceph mon.c for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: mon.c calling monitor election 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: mon.a calling monitor election 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: fsmap 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: mgrmap e41: y(active, since 16s), standbys: x 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: overall HEALTH_OK 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: mon.c calling monitor election 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: mon.a calling monitor election 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: fsmap 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: mgrmap e41: y(active, since 16s), standbys: x 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: overall HEALTH_OK 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:04 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: mon.c calling monitor election 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: mon.a calling monitor election 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: fsmap 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: mgrmap e41: y(active, since 16s), standbys: x 2024-02-13T15:15:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: overall HEALTH_OK 2024-02-13T15:15:04.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:04.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:04.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:04 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:06.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:06 smithi002 ceph-mon[64469]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:06.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:06 smithi002 ceph-mon[67023]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:06.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:06 smithi098 ceph-mon[25694]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T15:15:07.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:15:07.410+0000 7fe20d50d700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-13T15:15:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:08 smithi098 ceph-mon[25694]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:08 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:08 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:08.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:08.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[64469]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:08.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:08.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:08.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[67023]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:08.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:08.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:08 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:09 smithi098 ceph-mon[25694]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:10.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:09 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.036 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:09 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.078 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:09] "GET /metrics HTTP/1.1" 200 38171 "" "Prometheus/2.33.4" 2024-02-13T15:15:10.078 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:10.079 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.079 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.079 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:10.079 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:10.079 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:09 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:10 smithi098 ceph-mon[25694]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:10 smithi002 ceph-mon[64469]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:11.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:10 smithi002 ceph-mon[67023]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:12.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:12.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:12.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:12.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:15:12.835 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:12 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-13T15:15:13.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:12 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-13T15:15:13.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:13 smithi098 ceph-mon[25694]: Upgrade: It appears safe to stop mon.b 2024-02-13T15:15:13.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:13 smithi098 ceph-mon[25694]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:13.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:13 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:13.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:13 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:13.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:13 smithi098 ceph-mon[25694]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[64469]: Upgrade: It appears safe to stop mon.b 2024-02-13T15:15:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:14.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[67023]: Upgrade: It appears safe to stop mon.b 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:14.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:13 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:14.783 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:14 smithi098 ceph-mon[25694]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:14.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:14 smithi098 ceph-mon[25694]: Upgrade: Updating mon.b 2024-02-13T15:15:14.784 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:14 smithi098 ceph-mon[25694]: Deploying daemon mon.b on smithi098 2024-02-13T15:15:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[64469]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[64469]: Upgrade: Updating mon.b 2024-02-13T15:15:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[64469]: Deploying daemon mon.b on smithi098 2024-02-13T15:15:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[67023]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[67023]: Upgrade: Updating mon.b 2024-02-13T15:15:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:14 smithi002 ceph-mon[67023]: Deploying daemon mon.b on smithi098 2024-02-13T15:15:15.577 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:15 smithi098 systemd[1]: Stopping Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:15:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b[25677]: 2024-02-13T15:15:15.573+0000 7f172f78e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:15:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:15 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b[25677]: 2024-02-13T15:15:15.574+0000 7f172f78e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-13T15:15:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:15 smithi098 podman[57872]: 2024-02-13 15:15:15.768573105 +0000 UTC m=+0.304593207 container died 42fd2168af81f9e1a3f0ad0b1773bfcc64858b7cc037cc66c6f75b5d83d35ba0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, release=754, io.buildah.version=1.19.8, vcs-type=git, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, version=8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream) 2024-02-13T15:15:15.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE Bus STOPPING 2024-02-13T15:15:15.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:15:15.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE Bus STOPPED 2024-02-13T15:15:15.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE Bus STARTING 2024-02-13T15:15:15.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE Serving on http://:::9283 2024-02-13T15:15:15.877 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:15 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:15:15] ENGINE Bus STARTED 2024-02-13T15:15:16.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 podman[57872]: 2024-02-13 15:15:16.158182587 +0000 UTC m=+0.694202694 container cleanup 42fd2168af81f9e1a3f0ad0b1773bfcc64858b7cc037cc66c6f75b5d83d35ba0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, distribution-scope=public, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64) 2024-02-13T15:15:16.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 bash[57872]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b 2024-02-13T15:15:16.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 podman[57886]: 2024-02-13 15:15:16.41782418 +0000 UTC m=+0.640961306 container remove 42fd2168af81f9e1a3f0ad0b1773bfcc64858b7cc037cc66c6f75b5d83d35ba0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, release=754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., ceph=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-13T15:15:16.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-13T15:15:16.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 systemd[1]: Stopped Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:15:16.775 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.b.service: Consumed 9.458s CPU time. 2024-02-13T15:15:17.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 systemd[1]: Starting Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:15:17.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:16 smithi098 podman[58022]: 2024-02-13 15:15:16.892654459 +0000 UTC m=+0.018013061 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:15:17.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 podman[58022]: 2024-02-13 15:15:17.010435436 +0000 UTC m=+0.135794023 container create aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:15:17.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 podman[58022]: 2024-02-13 15:15:17.230024517 +0000 UTC m=+0.355383105 container init aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T15:15:17.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 podman[58022]: 2024-02-13 15:15:17.233470029 +0000 UTC m=+0.358828620 container start aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 2 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: pidfile_write: ignore empty --pid-file 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: load: jerasure load: lrc 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: RocksDB version: 7.9.2 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Git sha 0 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: DB SUMMARY 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: DB Session ID: 6X28M7PNCXABDD7FPM5W 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: CURRENT file: CURRENT 2024-02-13T15:15:17.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 981720 ; 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.error_if_exists: 0 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.create_if_missing: 0 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.paranoid_checks: 1 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.env: 0x560f002b0ae0 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T15:15:17.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.info_log: 0x560f029278c0 2024-02-13T15:15:17.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T15:15:17.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.statistics: (nil) 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.use_fsync: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_log_file_size: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_fallocate: 1 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T15:15:17.586 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.use_direct_reads: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.db_log_dir: 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.wal_dir: 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.write_buffer_manager: 0x560f028fb400 2024-02-13T15:15:17.587 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.unordered_write: 0 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.row_cache: None 2024-02-13T15:15:17.588 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.wal_filter: None 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.two_write_queues: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.wal_compression: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.atomic_flush: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.log_readahead_size: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T15:15:17.589 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_background_jobs: 2 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_background_compactions: -1 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_subcompactions: 1 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T15:15:17.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_open_files: -1 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_background_flushes: -1 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Compression algorithms supported: 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kZSTD supported: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kXpressCompression supported: 0 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T15:15:17.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kZlibCompression supported: 1 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kSnappyCompression supported: 1 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kLZ4Compression supported: 1 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: kBZip2Compression supported: 0 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.merge_operator: 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_filter: None 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T15:15:17.592 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f029279c0) 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_type: 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_index_type: 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_shortening: 1 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: checksum: 4 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: no_block_cache: 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache: 0x560f0292b090 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_options: 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: capacity : 536870912 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_shard_bits : 4 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: strict_capacity_limit : 0 2024-02-13T15:15:17.593 INFO:journalctl@ceph.mon.b.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_cache_compressed: (nil) 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: persistent_cache: (nil) 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size: 4096 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_size_deviation: 10 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_restart_interval: 16 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: index_block_restart_interval: 1 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: metadata_block_size: 4096 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: partition_filters: 0 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: use_delta_encoding: 1 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: filter_policy: bloomfilter 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: whole_key_filtering: 1 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: verify_compression: 0 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: format_version: 5 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_index_compression: 1 2024-02-13T15:15:17.594 INFO:journalctl@ceph.mon.b.smithi098.stdout: block_align: 0 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout: max_auto_readahead_size: 262144 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout: prepopulate_block_cache: 0 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout: initial_auto_readahead_size: 8192 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression: NoCompression 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.num_levels: 7 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T15:15:17.595 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T15:15:17.596 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T15:15:17.597 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.table_properties_collectors: 2024-02-13T15:15:17.598 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.inplace_update_support: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.bloom_locality: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.max_successive_merges: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.ttl: 2592000 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enable_blob_files: false 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.min_blob_size: 0 2024-02-13T15:15:17.599 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15074, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5de4a477-2028-4115-9eaa-b49cb6c3471a 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837317262662, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837317265905, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 584508, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15079, "largest_seqno": 15722, "table_properties": {"data_size": 581268, "index_size": 1137, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 8532, "raw_average_key_size": 20, "raw_value_size": 574100, "raw_average_value_size": 1350, "num_data_blocks": 51, "num_entries": 425, "num_filter_entries": 425, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707837317, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5de4a477-2028-4115-9eaa-b49cb6c3471a", "db_session_id": "6X28M7PNCXABDD7FPM5W", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-13T15:15:17.600 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707837317265990, "job": 1, "event": "recovery_finished"} 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560f02a1c000 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: DB pointer 0x560f02a06000 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** DB Stats ** 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T15:15:17.601 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: L0 1/0 570.81 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 310.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: L6 1/0 11.53 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: Sum 2/0 12.08 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 310.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 310.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** Compaction Stats [default] ** 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 310.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T15:15:17.602 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Cumulative compaction: 0.00 GB write, 72.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Interval compaction: 0.00 GB write, 72.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache BinnedLRUCache@0x560f0292b090#2 capacity: 512.00 MB usage: 2.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: starting mon.b rank 2 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???) e3 preinit fsid c0fca16a-ca80-11ee-95b9-87774f69a715 2024-02-13T15:15:17.603 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).mds e1 new map 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).mds e1 print_map 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: e1 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: legacy client fscid: -1 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout: No filesystems configured 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 ceph-mon[58041]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 bash[58022]: aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e 2024-02-13T15:15:17.604 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:17 smithi098 systemd[1]: Started Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:15:19.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:20.382 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:19] "GET /metrics HTTP/1.1" 200 38171 "" "Prometheus/2.33.4" 2024-02-13T15:15:22.628 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:15:22.276+0000 7fe20d50d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-13T15:15:24.525 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mon.c calling monitor election 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mon.b calling monitor election 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:15:25.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: overall HEALTH_OK 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mon.a calling monitor election 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mon.c calling monitor election 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: monmap e4: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: fsmap 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: mgrmap e41: y(active, since 37s), standbys: x 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: overall HEALTH_OK 2024-02-13T15:15:25.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='' 2024-02-13T15:15:25.085 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:25.085 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:24 smithi098 ceph-mon[58041]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mon.c calling monitor election 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mon.b calling monitor election 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: overall HEALTH_OK 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mon.a calling monitor election 2024-02-13T15:15:25.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mon.c calling monitor election 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: monmap e4: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: fsmap 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: mgrmap e41: y(active, since 37s), standbys: x 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: overall HEALTH_OK 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='' 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[64469]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mon.c calling monitor election 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mon.b calling monitor election 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: overall HEALTH_OK 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mon.a calling monitor election 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mon.c calling monitor election 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: monmap e4: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: fsmap 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: osdmap e104: 8 total, 8 up, 8 in 2024-02-13T15:15:25.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: mgrmap e41: y(active, since 37s), standbys: x 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: overall HEALTH_OK 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='' 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:25.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:24 smithi002 ceph-mon[67023]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 17s ago 7m 17.8M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (104s) 33s ago 6m 45.0M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (73s) 17s ago 6m 51.1M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (63s) 33s ago 11m 447M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (98s) 17s ago 13m 493M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (48s) 17s ago 13m 38.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:15:25.198 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 starting - - - 2048M 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (23s) 17s ago 12m 18.1M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (2m) 17s ago 7m 17.9M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (119s) 33s ago 7m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (11m) 17s ago 11m 53.2M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (11m) 17s ago 11m 53.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (10m) 17s ago 10m 49.1M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (10m) 17s ago 10m 53.8M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (9m) 33s ago 9m 53.3M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (9m) 33s ago 9m 50.8M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (9m) 33s ago 9m 49.5M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (8m) 33s ago 8m 51.7M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (69s) 33s ago 7m 55.8M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (6m) 17s ago 6m 73.6M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (6m) 33s ago 6m 74.3M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (6m) 17s ago 6m 71.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:15:25.199 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (6m) 33s ago 6m 74.2M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:15:25.606 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 5 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:15:25.607 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[67023]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[67023]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[67023]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2103453224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[64469]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[64469]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[64469]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:25.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:25 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2103453224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-02-13T15:15:25.947 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:15:25.948 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "4/23 daemons upgraded", 2024-02-13T15:15:25.948 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mon daemons", 2024-02-13T15:15:25.948 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:15:25.948 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:15:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:25 smithi098 ceph-mon[58041]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:25 smithi098 ceph-mon[58041]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:25 smithi098 ceph-mon[58041]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:25 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2103453224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:26.352 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:15:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:26 smithi098 ceph-mon[58041]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:26 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/763247979' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:15:27.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:26 smithi002 ceph-mon[64469]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:27.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:26 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/763247979' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:15:27.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:26 smithi002 ceph-mon[67023]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:27.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:26 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/763247979' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:15:28.496 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:28 smithi098 ceph-mon[58041]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:28.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:28 smithi002 ceph-mon[64469]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:28.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:28 smithi002 ceph-mon[67023]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:28.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:29.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:29 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:29.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:29 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:29.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:29 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:29.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:29.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:29.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:29.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:29.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:29.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:29] "GET /metrics HTTP/1.1" 200 38245 "" "Prometheus/2.33.4" 2024-02-13T15:15:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:30 smithi098 ceph-mon[58041]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:30 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:30 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[64469]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:30.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[67023]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:30.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:30 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:32.813 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:32 smithi098 ceph-mon[58041]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:32.814 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:32.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:32 smithi002 ceph-mon[64469]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:32.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:32 smithi002 ceph-mon[67023]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:32.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: Reconfiguring mon.a (monmap changed)... 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:33 smithi098 ceph-mon[58041]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: Reconfiguring mon.a (monmap changed)... 2024-02-13T15:15:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[64469]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:15:34.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:34.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: Reconfiguring mon.a (monmap changed)... 2024-02-13T15:15:34.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:34.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:34.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:34.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:33 smithi002 ceph-mon[67023]: Reconfiguring daemon mon.a on smithi002 2024-02-13T15:15:35.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: Reconfiguring mgr.y (monmap changed)... 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:35.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[64469]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: Reconfiguring mgr.y (monmap changed)... 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:35.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:35 smithi002 ceph-mon[67023]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:15:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: Reconfiguring mgr.y (monmap changed)... 2024-02-13T15:15:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:36.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:36.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:36.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:35 smithi098 ceph-mon[58041]: Reconfiguring daemon mgr.y on smithi002 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:36.944 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.945 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:36.945 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:36.945 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:36.945 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:36 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:36 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:37.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:37 smithi002 ceph-mon[64469]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:15:37.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:37 smithi002 ceph-mon[64469]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:15:37.970 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:37 smithi002 ceph-mon[67023]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:15:37.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:37 smithi002 ceph-mon[67023]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:15:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:37 smithi098 ceph-mon[58041]: Reconfiguring mon.c (monmap changed)... 2024-02-13T15:15:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:37 smithi098 ceph-mon[58041]: Reconfiguring daemon mon.c on smithi002 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[64469]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:38.868 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[67023]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:38.869 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:38.869 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:38.869 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:15:38.869 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:38 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-mon[58041]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:15:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:39.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[64469]: Reconfiguring osd.0 (monmap changed)... 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.0 on smithi002 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:39.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[67023]: Reconfiguring osd.0 (monmap changed)... 2024-02-13T15:15:39.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.0 on smithi002 2024-02-13T15:15:39.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:39 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:39 smithi098 ceph-mon[58041]: Reconfiguring osd.0 (monmap changed)... 2024-02-13T15:15:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:39 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.0 on smithi002 2024-02-13T15:15:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:39 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:40.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:39] "GET /metrics HTTP/1.1" 200 38242 "" "Prometheus/2.33.4" 2024-02-13T15:15:40.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:40.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:40.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: Reconfiguring osd.1 (monmap changed)... 2024-02-13T15:15:40.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:15:40.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.1 on smithi002 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: Reconfiguring osd.1 (monmap changed)... 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:40.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:40 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.1 on smithi002 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: Reconfiguring osd.1 (monmap changed)... 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:40 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.1 on smithi002 2024-02-13T15:15:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: Reconfiguring osd.2 (monmap changed)... 2024-02-13T15:15:42.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:15:42.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:42.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:42 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.2 on smithi002 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: Reconfiguring osd.2 (monmap changed)... 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.2 on smithi002 2024-02-13T15:15:42.601 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: Reconfiguring osd.2 (monmap changed)... 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:42.602 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:42 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.2 on smithi002 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: Reconfiguring osd.3 (monmap changed)... 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:44 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.3 on smithi002 2024-02-13T15:15:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: Reconfiguring osd.3 (monmap changed)... 2024-02-13T15:15:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.3 on smithi002 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: Reconfiguring osd.3 (monmap changed)... 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:44 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.3 on smithi002 2024-02-13T15:15:46.015 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: Reconfiguring rgw.foo.smithi002.jwgbut (monmap changed)... 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.016 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: Reconfiguring rgw.foo.smithi002.jwgbut (monmap changed)... 2024-02-13T15:15:46.017 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.017 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.017 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:45 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: Reconfiguring rgw.foo.smithi002.jwgbut (monmap changed)... 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:46.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:45 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: Reconfiguring daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.017 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: Reconfiguring daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.018 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:46 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: Reconfiguring daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:46 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:47.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[64469]: Reconfiguring rgw.smpl.smithi002.szwgvn (monmap changed)... 2024-02-13T15:15:47.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[64469]: Reconfiguring daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:15:47.997 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:47.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[67023]: Reconfiguring rgw.smpl.smithi002.szwgvn (monmap changed)... 2024-02-13T15:15:47.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[67023]: Reconfiguring daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:15:47.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:47 smithi098 ceph-mon[58041]: Reconfiguring rgw.smpl.smithi002.szwgvn (monmap changed)... 2024-02-13T15:15:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:47 smithi098 ceph-mon[58041]: Reconfiguring daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:15:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:15:48.966 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:48.967 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T15:15:49.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:49 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:50.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:49] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-02-13T15:15:50.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: Reconfiguring mon.b (monmap changed)... 2024-02-13T15:15:50.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: Reconfiguring daemon mon.b on smithi098 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T15:15:50.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:49 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:50 smithi098 ceph-mon[58041]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:50 smithi098 ceph-mon[58041]: Reconfiguring mgr.x (monmap changed)... 2024-02-13T15:15:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:50 smithi098 ceph-mon[58041]: Reconfiguring daemon mgr.x on smithi098 2024-02-13T15:15:51.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[64469]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[64469]: Reconfiguring mgr.x (monmap changed)... 2024-02-13T15:15:51.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[64469]: Reconfiguring daemon mgr.x on smithi098 2024-02-13T15:15:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[67023]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[67023]: Reconfiguring mgr.x (monmap changed)... 2024-02-13T15:15:51.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:50 smithi002 ceph-mon[67023]: Reconfiguring daemon mgr.x on smithi098 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: Reconfiguring osd.4 (monmap changed)... 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:15:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:52.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:52 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.4 on smithi098 2024-02-13T15:15:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: Reconfiguring osd.4 (monmap changed)... 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.4 on smithi098 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: Reconfiguring osd.4 (monmap changed)... 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:52.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:52 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.4 on smithi098 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: Reconfiguring osd.5 (monmap changed)... 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:54.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:53 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.5 on smithi098 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: Reconfiguring osd.5 (monmap changed)... 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.5 on smithi098 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: Reconfiguring osd.5 (monmap changed)... 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:54.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:53 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.5 on smithi098 2024-02-13T15:15:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: Reconfiguring osd.6 (monmap changed)... 2024-02-13T15:15:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:15:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:55.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:55 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.6 on smithi098 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: Reconfiguring osd.6 (monmap changed)... 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.6 on smithi098 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: Reconfiguring osd.6 (monmap changed)... 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:55.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:55 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.6 on smithi098 2024-02-13T15:15:56.725 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:15:56.776 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:56 smithi098 ceph-mon[58041]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:56.776 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.776 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.776 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:56 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:15:56.776 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:56 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[64469]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[67023]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:56.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:15:56.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:56 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:57.391 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 49s ago 7m 17.8M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (2m) 28s ago 7m 45.6M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (105s) 49s ago 6m 51.1M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (95s) 28s ago 12m 448M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (2m) 49s ago 13m 493M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (81s) 49s ago 13m 38.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (40s) 28s ago 12m 20.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (55s) 49s ago 12m 18.1M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (3m) 49s ago 8m 17.9M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (2m) 28s ago 8m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (11m) 49s ago 11m 53.2M 2737M 17.2.0 e1d6a67b021e 591e3c64150a 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (11m) 49s ago 11m 53.4M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:15:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (11m) 49s ago 11m 49.1M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (10m) 49s ago 10m 53.8M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (10m) 28s ago 10m 53.6M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (10m) 28s ago 10m 51.1M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (9m) 28s ago 9m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (9m) 28s ago 9m 52.3M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (101s) 28s ago 8m 58.1M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (6m) 49s ago 6m 73.6M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (6m) 28s ago 6m 74.6M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (6m) 49s ago 6m 71.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:15:57.393 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (6m) 28s ago 6m 74.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:15:57.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:57 smithi098 ceph-mon[58041]: Reconfiguring osd.7 (monmap changed)... 2024-02-13T15:15:57.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:57 smithi098 ceph-mon[58041]: Reconfiguring daemon osd.7 on smithi098 2024-02-13T15:15:57.717 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:57 smithi098 ceph-mon[58041]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:57.801 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:15:57.801 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:15:57.801 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:15:57.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:57.801 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 5 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:15:57.802 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[64469]: Reconfiguring osd.7 (monmap changed)... 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[64469]: Reconfiguring daemon osd.7 on smithi098 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[64469]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[67023]: Reconfiguring osd.7 (monmap changed)... 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[67023]: Reconfiguring daemon osd.7 on smithi098 2024-02-13T15:15:57.803 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:57 smithi002 ceph-mon[67023]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.148 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:15:58.148 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:15:58.148 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:15:58.149 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:15:58.149 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:15:58.149 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:15:58.149 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:15:58.149 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:15:58.150 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "5/23 daemons upgraded", 2024-02-13T15:15:58.150 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mon daemons", 2024-02-13T15:15:58.150 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:15:58.150 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:15:58.555 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:15:58.688 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:15:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:15:58.688 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1337627244' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.689 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:58 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1337627244' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1337627244' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:15:58.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:58 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:15:59.764 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:59 smithi098 ceph-mon[58041]: Reconfiguring rgw.foo.smithi098.eccznt (monmap changed)... 2024-02-13T15:15:59.764 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:59 smithi098 ceph-mon[58041]: Reconfiguring daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:15:59.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:59.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:59 smithi098 ceph-mon[58041]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:59.765 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:15:59 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1196713778' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[64469]: Reconfiguring rgw.foo.smithi098.eccznt (monmap changed)... 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[64469]: Reconfiguring daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[64469]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1196713778' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[67023]: Reconfiguring rgw.foo.smithi098.eccznt (monmap changed)... 2024-02-13T15:15:59.813 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[67023]: Reconfiguring daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:15:59.814 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:15:59.814 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[67023]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:15:59.814 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1196713778' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:16:00.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:15:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:15:59] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-02-13T15:16:00.710 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: Reconfiguring rgw.smpl.smithi098.kzxzji (monmap changed)... 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:00.711 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:00 smithi098 ceph-mon[58041]: Reconfiguring daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:16:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: Reconfiguring rgw.smpl.smithi098.kzxzji (monmap changed)... 2024-02-13T15:16:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[64469]: Reconfiguring daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: Reconfiguring rgw.smpl.smithi098.kzxzji (monmap changed)... 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:00 smithi002 ceph-mon[67023]: Reconfiguring daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:16:02.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all mon 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.584 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all crash 2024-02-13T15:16:02.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: Upgrade: osd.0 is safe to restart 2024-02-13T15:16:02.585 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:02.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all mon 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-13T15:16:02.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all crash 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: Upgrade: osd.0 is safe to restart 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:02.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all mon 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-13T15:16:02.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all crash 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-13T15:16:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: Upgrade: osd.0 is safe to restart 2024-02-13T15:16:02.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: Upgrade: Updating osd.0 2024-02-13T15:16:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[64469]: Deploying daemon osd.0 on smithi002 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: Upgrade: Updating osd.0 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:03 smithi002 ceph-mon[67023]: Deploying daemon osd.0 on smithi002 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: Upgrade: Updating osd.0 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:04.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:03 smithi098 ceph-mon[58041]: Deploying daemon osd.0 on smithi002 2024-02-13T15:16:06.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:05 smithi002 ceph-mon[64469]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:06.026 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:05 smithi002 ceph-mon[67023]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:06.288 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:06 smithi002 systemd[1]: Stopping Ceph osd.0 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:16:06.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:05 smithi098 ceph-mon[58041]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:06.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[32101]: 2024-02-13T15:16:06.286+0000 7f98fd452700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:16:06.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[32101]: 2024-02-13T15:16:06.286+0000 7f98fd452700 -1 osd.0 104 *** Got signal Terminated *** 2024-02-13T15:16:06.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:06 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[32101]: 2024-02-13T15:16:06.286+0000 7f98fd452700 -1 osd.0 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:16:07.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:06 smithi002 ceph-mon[64469]: osd.0 marked itself down and dead 2024-02-13T15:16:07.227 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:06 smithi002 ceph-mon[67023]: osd.0 marked itself down and dead 2024-02-13T15:16:07.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:06 smithi098 ceph-mon[58041]: osd.0 marked itself down and dead 2024-02-13T15:16:07.514 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:07 smithi002 podman[69936]: 2024-02-13 15:16:07.226539238 +0000 UTC m=+1.052510774 container died 591e3c64150a772318319f10cf45d898f8a33b5a796ce49e97de4fe7c9a507b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8) 2024-02-13T15:16:07.876 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:07 smithi002 podman[69936]: 2024-02-13 15:16:07.611694516 +0000 UTC m=+1.437666050 container cleanup 591e3c64150a772318319f10cf45d898f8a33b5a796ce49e97de4fe7c9a507b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.buildah.version=1.19.8, version=8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-13T15:16:07.876 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:07 smithi002 bash[69936]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0 2024-02-13T15:16:08.234 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:07 smithi002 podman[69954]: 2024-02-13 15:16:07.904934592 +0000 UTC m=+0.669675181 container remove 591e3c64150a772318319f10cf45d898f8a33b5a796ce49e97de4fe7c9a507b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, RELEASE=HEAD, version=8, architecture=x86_64, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-13T15:16:08.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:08.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: osdmap e105: 8 total, 7 up, 8 in 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[64469]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: osdmap e105: 8 total, 7 up, 8 in 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.235 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:07 smithi002 ceph-mon[67023]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: osdmap e105: 8 total, 7 up, 8 in 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:07 smithi098 ceph-mon[58041]: failed to encode map e105 with expected crc 2024-02-13T15:16:08.492 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.136102559 +0000 UTC m=+0.017765035 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:08.492 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.282389467 +0000 UTC m=+0.164051923 container create 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:16:08.492 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.492030872 +0000 UTC m=+0.373693342 container init 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:16:08.792 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.495350595 +0000 UTC m=+0.377013061 container start 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:16:08.792 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.583838277 +0000 UTC m=+0.465500738 container attach 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-13T15:16:09.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: osdmap e106: 8 total, 7 up, 8 in 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:08 smithi098 ceph-mon[58041]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.126 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:08 smithi002 podman[70046]: 2024-02-13 15:16:08.791724189 +0000 UTC m=+0.673386644 container died 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: osdmap e106: 8 total, 7 up, 8 in 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[64469]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: osdmap e106: 8 total, 7 up, 8 in 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:08 smithi002 ceph-mon[67023]: failed to encode map e106 with expected crc 2024-02-13T15:16:09.840 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 podman[70072]: 2024-02-13 15:16:09.387801726 +0000 UTC m=+0.587311107 container remove 3fe62a2ef9ac0abce03d363a8a71bfb490e0fc37b9a7b8d37aba2f2d5d4a277b (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-13T15:16:09.841 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-13T15:16:09.841 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 systemd[1]: Stopped Ceph osd.0 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:16:09.841 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.0.service: Consumed 6.993s CPU time. 2024-02-13T15:16:09.841 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 systemd[1]: Starting Ceph osd.0 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:16:10.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:09 smithi002 ceph-mon[64469]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:16:10.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:09 smithi002 ceph-mon[64469]: Health check failed: Reduced data availability: 14 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:10.093 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:09] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-02-13T15:16:10.094 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:09 smithi002 ceph-mon[67023]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:16:10.094 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:09 smithi002 ceph-mon[67023]: Health check failed: Reduced data availability: 14 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:10.094 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 podman[70175]: 2024-02-13 15:16:09.742153437 +0000 UTC m=+0.018977039 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:10.095 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:09 smithi002 podman[70175]: 2024-02-13 15:16:09.872912511 +0000 UTC m=+0.149736144 container create 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True) 2024-02-13T15:16:10.193 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:09 smithi098 ceph-mon[58041]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:16:10.193 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:09 smithi098 ceph-mon[58041]: Health check failed: Reduced data availability: 14 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:10.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 podman[70175]: 2024-02-13 15:16:10.09215915 +0000 UTC m=+0.368982752 container init 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-13T15:16:10.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 podman[70175]: 2024-02-13 15:16:10.095651809 +0000 UTC m=+0.372475419 container start 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:16:10.377 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 podman[70175]: 2024-02-13 15:16:10.192136021 +0000 UTC m=+0.468959620 container attach 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 bash[70175]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-13T15:16:10.872 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-13T15:16:10.873 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-13T15:16:10.873 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-13T15:16:10.873 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate[70190]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-13T15:16:10.873 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:10 smithi002 podman[70175]: 2024-02-13 15:16:10.564269206 +0000 UTC m=+0.841092796 container died 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-13T15:16:11.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 podman[70312]: 2024-02-13 15:16:11.179464563 +0000 UTC m=+0.605091454 container remove 1c942dc267b99b2eeaa356e5c0c94d469ec14b2e23b2581aa613c01785469fc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True) 2024-02-13T15:16:11.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 podman[70376]: 2024-02-13 15:16:11.299456327 +0000 UTC m=+0.018747308 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:11.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 podman[70376]: 2024-02-13 15:16:11.447230883 +0000 UTC m=+0.166521862 container create ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:16:11.759 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 podman[70376]: 2024-02-13 15:16:11.666070636 +0000 UTC m=+0.385361617 container init ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-13T15:16:11.759 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 podman[70376]: 2024-02-13 15:16:11.669098294 +0000 UTC m=+0.388389285 container start ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-13T15:16:11.759 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-osd[70395]: -- 172.21.15.2:0/2613589582 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5625322511e0 con 0x562532223400 2024-02-13T15:16:12.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[64469]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:16:12.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 bash[70376]: ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de 2024-02-13T15:16:12.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:11 smithi002 systemd[1]: Started Ceph osd.0 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:16:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[67023]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:16:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:11 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:11 smithi098 ceph-mon[58041]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:16:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:11 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:11 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:12.622 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:12 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:16:12.215+0000 7f62629557c0 -1 Falling back to public interface 2024-02-13T15:16:13.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:12 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:13.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:12 smithi098 ceph-mon[58041]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 57 pgs peering) 2024-02-13T15:16:13.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:12 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:13.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:12 smithi002 ceph-mon[64469]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 57 pgs peering) 2024-02-13T15:16:13.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:12 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 67/627 objects degraded (10.686%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:13.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:12 smithi002 ceph-mon[67023]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 57 pgs peering) 2024-02-13T15:16:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:13 smithi002 ceph-mon[64469]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:13 smithi002 ceph-mon[67023]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:14.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:13 smithi098 ceph-mon[58041]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:15.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:16:14.941+0000 7f62629557c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-13T15:16:15.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:16:14.968+0000 7f62629557c0 -1 osd.0 104 log_to_monitors true 2024-02-13T15:16:16.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:15 smithi002 ceph-mon[64469]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:16.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:15 smithi002 ceph-mon[64469]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:16:16.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:15 smithi002 ceph-mon[67023]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:16.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:15 smithi002 ceph-mon[67023]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:16:16.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:15 smithi098 ceph-mon[58041]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:16.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:15 smithi098 ceph-mon[58041]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T15:16:16.361 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:16:16.002+0000 7f6253dad700 -1 osd.0 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: osdmap e107: 8 total, 7 up, 8 in 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:17.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:16 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: osdmap e107: 8 total, 7 up, 8 in 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: osdmap e107: 8 total, 7 up, 8 in 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: failed to encode map e107 with expected crc 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:17.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:16 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109] boot 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: osdmap e108: 8 total, 8 up, 8 in 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.221 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109] boot 2024-02-13T15:16:18.222 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: osdmap e108: 8 total, 8 up, 8 in 2024-02-13T15:16:18.222 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:16:18.222 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.222 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: failed to encode map e107 with expected crc 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 67/627 objects degraded (10.686%) 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: failed to encode map e108 with expected crc 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: osd.0 [v2:172.21.15.2:6802/1988304109,v1:172.21.15.2:6803/1988304109] boot 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: osdmap e108: 8 total, 8 up, 8 in 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:18.984 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: osdmap e109: 8 total, 8 up, 8 in 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:18 smithi098 ceph-mon[58041]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: osdmap e109: 8 total, 8 up, 8 in 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[64469]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: osdmap e109: 8 total, 8 up, 8 in 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:18 smithi002 ceph-mon[67023]: failed to encode map e109 with expected crc 2024-02-13T15:16:19.986 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:19] "GET /metrics HTTP/1.1" 200 38373 "" "Prometheus/2.33.4" 2024-02-13T15:16:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:19 smithi098 ceph-mon[58041]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:19 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:20.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:19 smithi002 ceph-mon[64469]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:20.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:19 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:20.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:19 smithi002 ceph-mon[67023]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:20.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:19 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:21.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:21 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:16:21.393+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-13T15:16:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: pgmap v55: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:22 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-13T15:16:22.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: pgmap v55: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:22.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-13T15:16:22.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: pgmap v55: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:22.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:22 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-13T15:16:23.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:23 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-02-13T15:16:23.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:23 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:16:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:23 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-02-13T15:16:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:23 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:16:23.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:23 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-02-13T15:16:23.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:23 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:16:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:24 smithi098 ceph-mon[58041]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:16:24.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:24 smithi002 ceph-mon[64469]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:16:24.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:24 smithi002 ceph-mon[67023]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:16:26.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:26 smithi098 ceph-mon[58041]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:26.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:26 smithi002 ceph-mon[64469]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:26.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:26 smithi002 ceph-mon[67023]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:28.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:28 smithi098 ceph-mon[58041]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:16:28.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:28 smithi002 ceph-mon[64469]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:16:28.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:28 smithi002 ceph-mon[67023]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:16:28.928 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:16:29.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:28.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-13T15:16:29.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:29.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:29 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 13s ago 8m 26.2M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (2m) 61s ago 7m 45.6M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (2m) 13s ago 7m 53.5M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (2m) 61s ago 12m 448M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (2m) 13s ago 14m 500M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (113s) 13s ago 14m 45.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:16:29.596 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (72s) 61s ago 13m 20.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (87s) 13s ago 13m 37.7M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (3m) 13s ago 8m 19.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (3m) 61s ago 8m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (18s) 13s ago 12m 12.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (12m) 13s ago 12m 54.9M 2737M 17.2.0 e1d6a67b021e 2c3ee785a20c 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (11m) 13s ago 11m 50.5M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (11m) 13s ago 11m 55.1M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (10m) 61s ago 10m 53.6M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (10m) 61s ago 10m 51.1M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:16:29.597 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (10m) 61s ago 10m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (9m) 61s ago 9m 52.3M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (2m) 61s ago 9m 58.1M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (7m) 13s ago 7m 74.0M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (7m) 61s ago 7m 74.6M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (7m) 13s ago 7m 72.2M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:16:29.598 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (7m) 61s ago 7m 74.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:16:29.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:29 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:29.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:29 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:30.004 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 1 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:16:30.005 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:16:30.006 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-13T15:16:30.006 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 6 2024-02-13T15:16:30.006 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:16:30.006 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:16:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:29] "GET /metrics HTTP/1.1" 200 38373 "" "Prometheus/2.33.4" 2024-02-13T15:16:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "6/23 daemons upgraded", 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:16:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:16:30.546 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:30 smithi098 ceph-mon[58041]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-13T15:16:30.546 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:30 smithi098 ceph-mon[58041]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.546 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:30 smithi098 ceph-mon[58041]: from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:30 smithi098 ceph-mon[58041]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.547 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:30 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3571776402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:30.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[67023]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-13T15:16:30.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[67023]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[67023]: from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[67023]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3571776402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:30.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[64469]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-13T15:16:30.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[64469]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[64469]: from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[64469]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:30.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:30 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3571776402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:30.753 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:16:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:31 smithi002 ceph-mon[64469]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:31 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2903623953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:16:31.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:31 smithi002 ceph-mon[67023]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:31.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:31 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2903623953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:16:31.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:31 smithi098 ceph-mon[58041]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:16:31.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:31 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2903623953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:16:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:32 smithi098 ceph-mon[58041]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[64469]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[67023]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:34.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:34 smithi098 ceph-mon[58041]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:34.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:34 smithi002 ceph-mon[64469]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:34.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:34 smithi002 ceph-mon[67023]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:36.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:36 smithi098 ceph-mon[58041]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:36.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:36 smithi002 ceph-mon[64469]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:36.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:36 smithi002 ceph-mon[67023]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:37 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:37 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:37 smithi098 ceph-mon[58041]: Upgrade: osd.1 is safe to restart 2024-02-13T15:16:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[64469]: Upgrade: osd.1 is safe to restart 2024-02-13T15:16:37.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-13T15:16:37.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:37 smithi002 ceph-mon[67023]: Upgrade: osd.1 is safe to restart 2024-02-13T15:16:38.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-mon[58041]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:38.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:38.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:16:38.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:38.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[64469]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:38.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:38.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:16:38.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:38.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[67023]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:38.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:38.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T15:16:38.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:38 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:39.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-13T15:16:39.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[64469]: Upgrade: Updating osd.1 2024-02-13T15:16:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[64469]: Deploying daemon osd.1 on smithi002 2024-02-13T15:16:39.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[67023]: Upgrade: Updating osd.1 2024-02-13T15:16:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-mon[67023]: Deploying daemon osd.1 on smithi002 2024-02-13T15:16:39.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:39 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:39.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:39 smithi098 ceph-mon[58041]: Upgrade: Updating osd.1 2024-02-13T15:16:39.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:39 smithi098 ceph-mon[58041]: Deploying daemon osd.1 on smithi002 2024-02-13T15:16:40.035 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:39] "GET /metrics HTTP/1.1" 200 38381 "" "Prometheus/2.33.4" 2024-02-13T15:16:40.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:40 smithi002 ceph-mon[64469]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:40.562 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:40 smithi002 ceph-mon[67023]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:40.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:40 smithi098 ceph-mon[58041]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:16:41.828 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:41 smithi002 systemd[1]: Stopping Ceph osd.1 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:16:42.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[35060]: 2024-02-13T15:16:41.826+0000 7f7fc9b4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:16:42.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[35060]: 2024-02-13T15:16:41.826+0000 7f7fc9b4a700 -1 osd.1 109 *** Got signal Terminated *** 2024-02-13T15:16:42.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[35060]: 2024-02-13T15:16:41.826+0000 7f7fc9b4a700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:16:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:42 smithi098 ceph-mon[58041]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:42 smithi098 ceph-mon[58041]: osd.1 marked itself down and dead 2024-02-13T15:16:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:42 smithi002 ceph-mon[64469]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:42 smithi002 ceph-mon[64469]: osd.1 marked itself down and dead 2024-02-13T15:16:42.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:42 smithi002 ceph-mon[67023]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:42.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:42 smithi002 ceph-mon[67023]: osd.1 marked itself down and dead 2024-02-13T15:16:43.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:42 smithi002 podman[72393]: 2024-02-13 15:16:42.854508112 +0000 UTC m=+1.150431291 container died 2c3ee785a20c777d4a84bacb698f8abc94b22890633ca48035c490489ad5b6ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8) 2024-02-13T15:16:43.513 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:43 smithi002 podman[72393]: 2024-02-13 15:16:43.243694914 +0000 UTC m=+1.539618084 container cleanup 2c3ee785a20c777d4a84bacb698f8abc94b22890633ca48035c490489ad5b6ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, architecture=x86_64, distribution-scope=public, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-13T15:16:43.513 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:43 smithi002 bash[72393]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1 2024-02-13T15:16:43.513 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: osdmap e110: 8 total, 7 up, 8 in 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[64469]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.514 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: osdmap e110: 8 total, 7 up, 8 in 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.515 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:43 smithi002 ceph-mon[67023]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: osdmap e110: 8 total, 7 up, 8 in 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:43 smithi098 ceph-mon[58041]: failed to encode map e110 with expected crc 2024-02-13T15:16:43.806 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:43 smithi002 podman[72419]: 2024-02-13 15:16:43.51224787 +0000 UTC m=+0.650278754 container remove 2c3ee785a20c777d4a84bacb698f8abc94b22890633ca48035c490489ad5b6ae (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-02-13T15:16:44.100 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:43 smithi002 podman[72512]: 2024-02-13 15:16:43.708502589 +0000 UTC m=+0.016967831 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:44.101 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:43 smithi002 podman[72512]: 2024-02-13 15:16:43.830096263 +0000 UTC m=+0.138561553 container create 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-13T15:16:44.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 podman[72512]: 2024-02-13 15:16:44.099767202 +0000 UTC m=+0.408232444 container init 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:16:44.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 podman[72512]: 2024-02-13 15:16:44.103212948 +0000 UTC m=+0.411678181 container start 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-13T15:16:44.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 podman[72512]: 2024-02-13 15:16:44.165906244 +0000 UTC m=+0.474371481 container attach 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: pgmap v67: 161 pgs: 45 peering, 116 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: osdmap e111: 8 total, 7 up, 8 in 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[64469]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: pgmap v67: 161 pgs: 45 peering, 116 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: osdmap e111: 8 total, 7 up, 8 in 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:44 smithi002 ceph-mon[67023]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: pgmap v67: 161 pgs: 45 peering, 116 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: osdmap e111: 8 total, 7 up, 8 in 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:44 smithi098 ceph-mon[58041]: failed to encode map e111 with expected crc 2024-02-13T15:16:44.876 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 podman[72512]: 2024-02-13 15:16:44.413434074 +0000 UTC m=+0.721899320 container died 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:16:45.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 podman[72538]: 2024-02-13 15:16:44.894661041 +0000 UTC m=+0.469208544 container remove 3379d3aa179f1c7ead87e0539f05e7be1ed550542c2a4ed568a368aed794ad6e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:16:45.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-13T15:16:45.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 systemd[1]: Stopped Ceph osd.1 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:16:45.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:44 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.1.service: Consumed 7.731s CPU time. 2024-02-13T15:16:45.422 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 systemd[1]: Starting Ceph osd.1 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:16:45.422 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 podman[72638]: 2024-02-13 15:16:45.287640699 +0000 UTC m=+0.019422078 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:45.422 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 podman[72638]: 2024-02-13 15:16:45.421222532 +0000 UTC m=+0.153003907 container create 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-13T15:16:45.757 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 podman[72638]: 2024-02-13 15:16:45.665115913 +0000 UTC m=+0.396897290 container init 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-13T15:16:45.758 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 podman[72638]: 2024-02-13 15:16:45.668222466 +0000 UTC m=+0.400003844 container start 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-13T15:16:46.115 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:45 smithi002 podman[72638]: 2024-02-13 15:16:45.756996021 +0000 UTC m=+0.488777399 container attach 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:16:46.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-mon[64469]: pgmap v69: 161 pgs: 12 stale+active+clean, 45 peering, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:16:46.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-mon[67023]: pgmap v69: 161 pgs: 12 stale+active+clean, 45 peering, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:16:46.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 bash[72638]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate[72653]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-13T15:16:46.377 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 podman[72638]: 2024-02-13 15:16:46.138597166 +0000 UTC m=+0.870378536 container died 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-13T15:16:46.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:46 smithi098 ceph-mon[58041]: pgmap v69: 161 pgs: 12 stale+active+clean, 45 peering, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:16:47.096 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 podman[72790]: 2024-02-13 15:16:46.744748673 +0000 UTC m=+0.601042214 container remove 474309a86207806c983125392d93e05f37834dabd1b3f8663f13ab12dc1d3a35 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-13T15:16:47.096 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:46 smithi002 podman[72854]: 2024-02-13 15:16:46.866667617 +0000 UTC m=+0.018641266 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:16:47.349 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 podman[72854]: 2024-02-13 15:16:47.095762831 +0000 UTC m=+0.247736463 container create c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-13T15:16:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 podman[72854]: 2024-02-13 15:16:47.34828148 +0000 UTC m=+0.500255113 container init c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:16:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 podman[72854]: 2024-02-13 15:16:47.351469279 +0000 UTC m=+0.503442920 container start c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True) 2024-02-13T15:16:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-osd[72873]: -- 172.21.15.2:0/3963404604 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de51e8b1e0 con 0x55de51e5d000 2024-02-13T15:16:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 bash[72854]: c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 2024-02-13T15:16:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 systemd[1]: Started Ceph osd.1 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:16:48.312 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:16:47.895+0000 7f7a4b9087c0 -1 Falling back to public interface 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: pgmap v70: 161 pgs: 11 stale+active+clean, 1 active+undersized+degraded, 45 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: pgmap v70: 161 pgs: 11 stale+active+clean, 1 active+undersized+degraded, 45 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-02-13T15:16:48.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.314 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: pgmap v70: 161 pgs: 11 stale+active+clean, 1 active+undersized+degraded, 45 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-02-13T15:16:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:16:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-02-13T15:16:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:48 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:49.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:49.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:48 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:49.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:48 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:50.184 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:49 smithi002 ceph-mon[64469]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:50.184 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:49 smithi002 ceph-mon[64469]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-02-13T15:16:50.185 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:49] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-02-13T15:16:50.185 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:49 smithi002 ceph-mon[67023]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:50.185 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:49 smithi002 ceph-mon[67023]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-02-13T15:16:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:49 smithi098 ceph-mon[58041]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:49 smithi098 ceph-mon[58041]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-02-13T15:16:50.876 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:16:50.614+0000 7f7a4b9087c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-13T15:16:50.876 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:16:50.647+0000 7f7a4b9087c0 -1 osd.1 109 log_to_monitors true 2024-02-13T15:16:51.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-mon[64469]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-mon[64469]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.204 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-mon[67023]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.204 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:50 smithi002 ceph-mon[67023]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:50 smithi098 ceph-mon[58041]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:50 smithi098 ceph-mon[58041]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T15:16:51.486 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:16:51.202+0000 7f7a38d58700 -1 osd.1 109 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: failed to encode map e112 with expected crc 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: osdmap e112: 8 total, 7 up, 8 in 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[67023]: from='osd.1 ' entity='osd.1' 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: failed to encode map e112 with expected crc 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: osdmap e112: 8 total, 7 up, 8 in 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:51 smithi002 ceph-mon[64469]: from='osd.1 ' entity='osd.1' 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: failed to encode map e112 with expected crc 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: osdmap e112: 8 total, 7 up, 8 in 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: from='osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:16:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:51 smithi098 ceph-mon[58041]: from='osd.1 ' entity='osd.1' 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: failed to encode map e113 with expected crc 2024-02-13T15:16:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078] boot 2024-02-13T15:16:53.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: osdmap e113: 8 total, 8 up, 8 in 2024-02-13T15:16:53.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: failed to encode map e113 with expected crc 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078] boot 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: osdmap e113: 8 total, 8 up, 8 in 2024-02-13T15:16:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: failed to encode map e112 with expected crc 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: failed to encode map e113 with expected crc 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: osd.1 [v2:172.21.15.2:6810/1562153078,v1:172.21.15.2:6811/1562153078] boot 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: osdmap e113: 8 total, 8 up, 8 in 2024-02-13T15:16:53.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: pgmap v75: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: osdmap e114: 8 total, 8 up, 8 in 2024-02-13T15:16:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:54 smithi098 ceph-mon[58041]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: pgmap v75: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-02-13T15:16:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: osdmap e114: 8 total, 8 up, 8 in 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[67023]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: pgmap v75: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded (PG_DEGRADED) 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: osdmap e114: 8 total, 8 up, 8 in 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:54 smithi002 ceph-mon[64469]: failed to encode map e114 with expected crc 2024-02-13T15:16:56.476 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:56 smithi002 ceph-mon[64469]: pgmap v77: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-02-13T15:16:56.476 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:56 smithi002 ceph-mon[67023]: pgmap v77: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-02-13T15:16:56.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:56 smithi098 ceph-mon[58041]: pgmap v77: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-02-13T15:16:56.876 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:16:56.681+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-13T15:16:57.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:57 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-13T15:16:57.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:16:57.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-13T15:16:57.879 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:57 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:16:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:58 smithi098 ceph-mon[58041]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:58 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded) 2024-02-13T15:16:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:58 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:16:58.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-13T15:16:58.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:16:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:16:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[64469]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded) 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[67023]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 8 pgs degraded) 2024-02-13T15:16:58.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:58 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:16:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:16:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:59.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:16:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:16:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:16:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:00.282 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:16:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:16:59] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-02-13T15:17:00.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:00 smithi002 ceph-mon[64469]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T15:17:00.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:00 smithi002 ceph-mon[67023]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T15:17:00.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:00 smithi098 ceph-mon[58041]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T15:17:01.116 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 9s ago 8m 26.2M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (3m) 93s ago 8m 45.6M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (2m) 9s ago 7m 53.7M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (2m) 93s ago 13m 448M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:17:01.788 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (3m) 9s ago 14m 502M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (2m) 9s ago 15m 47.3M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (104s) 93s ago 13m 20.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (119s) 9s ago 13m 40.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 9s ago 9m 19.1M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (3m) 93s ago 9m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (50s) 9s ago 13m 42.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (14s) 9s ago 12m 14.3M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (12m) 9s ago 12m 52.3M 2737M 17.2.0 e1d6a67b021e aa71a56938cc 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (11m) 9s ago 11m 56.7M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (11m) 93s ago 11m 53.6M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (11m) 93s ago 11m 51.1M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (10m) 93s ago 10m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (10m) 93s ago 10m 52.3M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:17:01.789 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (2m) 93s ago 9m 58.1M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:17:01.790 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (7m) 9s ago 7m 74.2M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:17:01.790 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (7m) 93s ago 7m 74.6M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:17:01.790 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (7m) 9s ago 7m 72.4M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:17:01.790 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (7m) 93s ago 7m 74.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:17:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 7 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:17:02.195 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "7/23 daemons upgraded", 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:17:02.538 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:02 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/820623671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/820623671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:02.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:02 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/820623671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:02.941 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:17:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:03 smithi098 ceph-mon[58041]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:03 smithi098 ceph-mon[58041]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:03 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1137029139' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:03.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[64469]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[64469]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1137029139' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:03.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[67023]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[67023]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:03.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:03 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1137029139' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:04 smithi098 ceph-mon[58041]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:17:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:04 smithi002 ceph-mon[64469]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:17:04.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:04 smithi002 ceph-mon[67023]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:17:06.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:06 smithi098 ceph-mon[58041]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-02-13T15:17:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:06 smithi002 ceph-mon[64469]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-02-13T15:17:06.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:06 smithi002 ceph-mon[67023]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-02-13T15:17:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:07 smithi098 ceph-mon[58041]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:08.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:07 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:08.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:07 smithi002 ceph-mon[64469]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:08.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:07 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:07 smithi002 ceph-mon[67023]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:08.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:07 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:08.948 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-13T15:17:08.948 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:09.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:08 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:09.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:08 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:09.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:08 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:09 smithi002 ceph-mon[64469]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:10.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:09] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.33.4" 2024-02-13T15:17:10.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:09 smithi002 ceph-mon[67023]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:10.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:09 smithi098 ceph-mon[58041]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[64469]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[64469]: Upgrade: osd.2 is safe to restart 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[67023]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.305 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.306 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:11 smithi002 ceph-mon[67023]: Upgrade: osd.2 is safe to restart 2024-02-13T15:17:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:11 smithi098 ceph-mon[58041]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:11 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:11 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-13T15:17:12.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:11 smithi098 ceph-mon[58041]: Upgrade: osd.2 is safe to restart 2024-02-13T15:17:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: Upgrade: Updating osd.2 2024-02-13T15:17:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:17:14.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:14.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:14 smithi098 ceph-mon[58041]: Deploying daemon osd.2 on smithi002 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: Upgrade: Updating osd.2 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:14.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[64469]: Deploying daemon osd.2 on smithi002 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: Upgrade: Updating osd.2 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:14.590 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:14 smithi002 ceph-mon[67023]: Deploying daemon osd.2 on smithi002 2024-02-13T15:17:16.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:16 smithi002 ceph-mon[64469]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:16.541 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:16 smithi002 ceph-mon[67023]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:16.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:16 smithi098 ceph-mon[58041]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:16.807 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:16 smithi002 systemd[1]: Stopping Ceph osd.2 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:17:17.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[37977]: 2024-02-13T15:17:16.805+0000 7f73560b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:17:17.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[37977]: 2024-02-13T15:17:16.805+0000 7f73560b7700 -1 osd.2 114 *** Got signal Terminated *** 2024-02-13T15:17:17.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[37977]: 2024-02-13T15:17:16.805+0000 7f73560b7700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:17:17.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:17 smithi098 ceph-mon[58041]: osd.2 marked itself down and dead 2024-02-13T15:17:17.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:17.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:17 smithi002 ceph-mon[64469]: osd.2 marked itself down and dead 2024-02-13T15:17:17.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:17 smithi002 ceph-mon[67023]: osd.2 marked itself down and dead 2024-02-13T15:17:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:17.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:17 smithi002 podman[74865]: 2024-02-13 15:17:17.678960948 +0000 UTC m=+0.989887873 container died aa71a56938cc294655b46cee282b0a24818cb3841d869e819185cd16f1e30087 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, release=754, vcs-type=git, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_CLEAN=True, distribution-scope=public, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-13T15:17:18.284 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74865]: 2024-02-13 15:17:18.063850982 +0000 UTC m=+1.374777895 container cleanup aa71a56938cc294655b46cee282b0a24818cb3841d869e819185cd16f1e30087 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, ceph=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-13T15:17:18.285 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 bash[74865]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: osdmap e115: 8 total, 7 up, 8 in 2024-02-13T15:17:18.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-mon[58041]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.623 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74893]: 2024-02-13 15:17:18.306552626 +0000 UTC m=+0.620629921 container remove aa71a56938cc294655b46cee282b0a24818cb3841d869e819185cd16f1e30087 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-type=git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: osdmap e115: 8 total, 7 up, 8 in 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[67023]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: osdmap e115: 8 total, 7 up, 8 in 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:18 smithi002 ceph-mon[64469]: failed to encode map e115 with expected crc 2024-02-13T15:17:18.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74989]: 2024-02-13 15:17:18.526684173 +0000 UTC m=+0.018499026 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:18.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74989]: 2024-02-13 15:17:18.641621727 +0000 UTC m=+0.133436576 container create 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True) 2024-02-13T15:17:18.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74989]: 2024-02-13 15:17:18.86860288 +0000 UTC m=+0.360417731 container init 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-13T15:17:18.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74989]: 2024-02-13 15:17:18.872361345 +0000 UTC m=+0.364176203 container start 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:17:19.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:19.289 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:18 smithi002 podman[74989]: 2024-02-13 15:17:18.979368972 +0000 UTC m=+0.471183830 container attach 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-13T15:17:19.289 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:19 smithi002 podman[74989]: 2024-02-13 15:17:19.158418109 +0000 UTC m=+0.650232969 container died 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: osdmap e116: 8 total, 7 up, 8 in 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:19 smithi098 ceph-mon[58041]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: osdmap e116: 8 total, 7 up, 8 in 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[64469]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:19.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: osdmap e116: 8 total, 7 up, 8 in 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:19.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-mon[67023]: failed to encode map e116 with expected crc 2024-02-13T15:17:20.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:19] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.33.4" 2024-02-13T15:17:20.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:19 smithi002 podman[75015]: 2024-02-13 15:17:19.808569798 +0000 UTC m=+0.645533372 container remove 7d9c962a8169f0a85ba7f71ec7320fa987748d31494ec3244084d9407fae83cd (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD) 2024-02-13T15:17:20.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:19 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-13T15:17:20.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:19 smithi002 systemd[1]: Stopped Ceph osd.2 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:17:20.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:19 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.2.service: Consumed 6.726s CPU time. 2024-02-13T15:17:20.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 systemd[1]: Starting Ceph osd.2 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:17:20.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-mon[64469]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:17:20.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-mon[64469]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:20.522 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-mon[67023]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:17:20.522 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-mon[67023]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:20.522 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.165186039 +0000 UTC m=+0.017887862 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:20.522 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.276320111 +0000 UTC m=+0.129021928 container create 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-13T15:17:20.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:20 smithi098 ceph-mon[58041]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:17:20.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:20 smithi098 ceph-mon[58041]: Health check failed: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:20.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.520830838 +0000 UTC m=+0.373532670 container init 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-13T15:17:20.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.524442546 +0000 UTC m=+0.377144372 container start 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-13T15:17:20.876 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.612462043 +0000 UTC m=+0.465163865 container attach 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 bash[75120]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-13T15:17:21.300 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-13T15:17:21.301 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate[75135]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-13T15:17:21.301 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:20 smithi002 podman[75120]: 2024-02-13 15:17:20.996557332 +0000 UTC m=+0.849259147 container died 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-02-13T15:17:21.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:21 smithi002 podman[75256]: 2024-02-13 15:17:21.599506382 +0000 UTC m=+0.594919966 container remove 9c5ff2545353e7014bea198e359a5253e0d8ab006db5a0a48062bd5fc82a1c3a (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-13T15:17:21.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:21 smithi002 podman[75320]: 2024-02-13 15:17:21.719685827 +0000 UTC m=+0.018169508 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:21.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:21 smithi002 podman[75320]: 2024-02-13 15:17:21.850644147 +0000 UTC m=+0.149127815 container create d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD) 2024-02-13T15:17:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[64469]: pgmap v92: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:22.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[67023]: pgmap v92: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:22.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:22.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:22.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 podman[75320]: 2024-02-13 15:17:22.111469124 +0000 UTC m=+0.409952807 container init d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-13T15:17:22.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 podman[75320]: 2024-02-13 15:17:22.114882123 +0000 UTC m=+0.413365795 container start d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-13T15:17:22.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-osd[75339]: -- 172.21.15.2:0/2132912373 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55697f5c51e0 con 0x55697f597000 2024-02-13T15:17:22.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 bash[75320]: d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 2024-02-13T15:17:22.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 systemd[1]: Started Ceph osd.2 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:17:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:22 smithi098 ceph-mon[58041]: pgmap v92: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:22.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:23.116 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:22 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:17:22.658+0000 7f32503eb7c0 -1 Falling back to public interface 2024-02-13T15:17:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:23 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:23 smithi002 ceph-mon[64469]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-02-13T15:17:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:23 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:23.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:23 smithi002 ceph-mon[67023]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-02-13T15:17:23.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:23 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:23.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:23 smithi098 ceph-mon[58041]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-02-13T15:17:24.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:24 smithi002 ceph-mon[64469]: pgmap v93: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:24.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:24 smithi002 ceph-mon[67023]: pgmap v93: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:24.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:24 smithi098 ceph-mon[58041]: pgmap v93: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:25.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:25 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:17:25.358+0000 7f32503eb7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-13T15:17:25.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:25 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:17:25.379+0000 7f32503eb7c0 -1 osd.2 114 log_to_monitors true 2024-02-13T15:17:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:26 smithi002 ceph-mon[64469]: pgmap v94: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:26 smithi002 ceph-mon[64469]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:17:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:26 smithi002 ceph-mon[67023]: pgmap v94: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:26 smithi002 ceph-mon[67023]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:17:26.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:26 smithi098 ceph-mon[58041]: pgmap v94: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:26.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:26 smithi098 ceph-mon[58041]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T15:17:27.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:17:26 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:17:26.655+0000 7f323d83b700 -1 osd.2 114 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: failed to encode map e117 with expected crc 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: osdmap e117: 8 total, 7 up, 8 in 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[64469]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: failed to encode map e117 with expected crc 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: osdmap e117: 8 total, 7 up, 8 in 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:27 smithi002 ceph-mon[67023]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' 2024-02-13T15:17:27.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T15:17:27.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: failed to encode map e117 with expected crc 2024-02-13T15:17:27.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: osdmap e117: 8 total, 7 up, 8 in 2024-02-13T15:17:27.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:17:27.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:27.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:27 smithi098 ceph-mon[58041]: from='osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437]' entity='osd.2' 2024-02-13T15:17:28.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: failed to encode map e117 with expected crc 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: pgmap v96: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437] boot 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: osdmap e118: 8 total, 8 up, 8 in 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: failed to encode map e117 with expected crc 2024-02-13T15:17:28.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: pgmap v96: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437] boot 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: osdmap e118: 8 total, 8 up, 8 in 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:28.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:28 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:28.663 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: failed to encode map e117 with expected crc 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: pgmap v96: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: osd.2 [v2:172.21.15.2:6818/1079710437,v1:172.21.15.2:6819/1079710437] boot 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: osdmap e118: 8 total, 8 up, 8 in 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:28.664 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:29.081 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:28 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:28.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: osdmap e119: 8 total, 8 up, 8 in 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: osdmap e119: 8 total, 8 up, 8 in 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e118 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: osdmap e119: 8 total, 8 up, 8 in 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:29.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:29 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:29] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.33.4" 2024-02-13T15:17:30.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[64469]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[64469]: pgmap v99: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:30.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:30.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[67023]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[67023]: pgmap v99: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:30.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:30 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:30 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:30 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:30 smithi098 ceph-mon[58041]: failed to encode map e119 with expected crc 2024-02-13T15:17:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:30 smithi098 ceph-mon[58041]: pgmap v99: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:30.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:30 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:31.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:31 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:17:31.369+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: pgmap v100: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:32.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: pgmap v100: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:17:32.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: pgmap v100: 161 pgs: 24 peering, 5 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:32.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:33.316 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:17:33.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:33 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-02-13T15:17:33.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:33 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:17:33.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:33 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-02-13T15:17:33.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:33 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:17:33.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:33 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-02-13T15:17:33.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:33 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:17:33.986 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:17:33.986 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (4m) 7s ago 9m 26.2M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:17:33.986 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (3m) 2m ago 9m 45.6M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:17:33.986 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (3m) 7s ago 8m 54.1M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (3m) 2m ago 13m 448M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (3m) 7s ago 15m 504M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (2m) 7s ago 15m 48.9M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (2m) 2m ago 14m 20.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (2m) 7s ago 14m 42.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 7s ago 10m 19.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (4m) 2m ago 9m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (82s) 7s ago 13m 43.5M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (47s) 7s ago 13m 50.2M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:17:33.987 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (12s) 7s ago 12m 16.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (12m) 7s ago 12m 57.5M 2737M 17.2.0 e1d6a67b021e eaf3a325b048 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (12m) 2m ago 11m 53.6M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (11m) 2m ago 11m 51.1M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (11m) 2m ago 11m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (10m) 2m ago 10m 52.3M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (3m) 2m ago 10m 58.1M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (8m) 7s ago 8m 74.3M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (8m) 2m ago 8m 74.6M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (8m) 7s ago 8m 72.7M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:17:33.988 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (8m) 2m ago 8m 74.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:17:34.390 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:17:34.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 8 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:17:34.392 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:17:34.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:34 smithi098 ceph-mon[58041]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:17:34.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:34 smithi098 ceph-mon[58041]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:34 smithi098 ceph-mon[58041]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[64469]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:17:34.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[64469]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[64469]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[67023]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-13T15:17:34.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[67023]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:34 smithi002 ceph-mon[67023]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:34.736 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:17:34.736 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:17:34.736 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:17:34.736 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:17:34.736 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "8/23 daemons upgraded", 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:17:34.737 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:17:35.143 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:17:35.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:35 smithi098 ceph-mon[58041]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:35 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1655625363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:35.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:35 smithi098 ceph-mon[58041]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:35 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2301879055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:35.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[67023]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1655625363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:35.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[67023]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2301879055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:35.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[64469]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1655625363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:17:35.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[64469]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:17:35.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:35 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2301879055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:17:36.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:36 smithi098 ceph-mon[58041]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:36.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:36 smithi002 ceph-mon[64469]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:36.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:36 smithi002 ceph-mon[67023]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:38.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:38 smithi098 ceph-mon[58041]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:17:38.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:38 smithi002 ceph-mon[64469]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:17:38.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:38 smithi002 ceph-mon[67023]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:17:39.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:39.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:39 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:39.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:39 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:39.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:39 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:40.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:39] "GET /metrics HTTP/1.1" 200 38480 "" "Prometheus/2.33.4" 2024-02-13T15:17:40.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:40 smithi098 ceph-mon[58041]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-02-13T15:17:40.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:40 smithi002 ceph-mon[64469]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-02-13T15:17:40.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:40 smithi002 ceph-mon[67023]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-02-13T15:17:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:42 smithi098 ceph-mon[58041]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:42 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:42 smithi002 ceph-mon[64469]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:42 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:42.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:42 smithi002 ceph-mon[67023]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:42.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:42 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:44.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:44 smithi098 ceph-mon[58041]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:44.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:44 smithi002 ceph-mon[64469]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:44.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:44 smithi002 ceph-mon[67023]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:46.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:46 smithi098 ceph-mon[58041]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:46.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:46 smithi002 ceph-mon[64469]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:46.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:46 smithi002 ceph-mon[67023]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:47 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:47 smithi098 ceph-mon[58041]: Upgrade: osd.3 is safe to restart 2024-02-13T15:17:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:47.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[64469]: Upgrade: osd.3 is safe to restart 2024-02-13T15:17:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-13T15:17:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[67023]: Upgrade: osd.3 is safe to restart 2024-02-13T15:17:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:47.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:17:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:48 smithi098 ceph-mon[58041]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:17:48.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[64469]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[67023]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T15:17:48.599 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:17:49.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[64469]: Upgrade: Updating osd.3 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[64469]: Deploying daemon osd.3 on smithi002 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[67023]: Upgrade: Updating osd.3 2024-02-13T15:17:49.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-mon[67023]: Deploying daemon osd.3 on smithi002 2024-02-13T15:17:49.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:49 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:49.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:49 smithi098 ceph-mon[58041]: Upgrade: Updating osd.3 2024-02-13T15:17:49.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:49 smithi098 ceph-mon[58041]: Deploying daemon osd.3 on smithi002 2024-02-13T15:17:50.040 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:49] "GET /metrics HTTP/1.1" 200 38480 "" "Prometheus/2.33.4" 2024-02-13T15:17:50.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:50 smithi098 ceph-mon[58041]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:50.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:50 smithi002 ceph-mon[64469]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:50.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:50 smithi002 ceph-mon[67023]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:17:51.649 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:51 smithi002 systemd[1]: Stopping Ceph osd.3 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:17:52.126 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[40964]: 2024-02-13T15:17:51.648+0000 7f075ac85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:17:52.126 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[40964]: 2024-02-13T15:17:51.648+0000 7f075ac85700 -1 osd.3 119 *** Got signal Terminated *** 2024-02-13T15:17:52.126 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:51 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[40964]: 2024-02-13T15:17:51.648+0000 7f075ac85700 -1 osd.3 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:17:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:52 smithi098 ceph-mon[58041]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:52 smithi098 ceph-mon[58041]: osd.3 marked itself down and dead 2024-02-13T15:17:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:52 smithi002 ceph-mon[64469]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:52 smithi002 ceph-mon[64469]: osd.3 marked itself down and dead 2024-02-13T15:17:52.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:52 smithi002 ceph-mon[67023]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:17:52.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:52 smithi002 ceph-mon[67023]: osd.3 marked itself down and dead 2024-02-13T15:17:53.008 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:52 smithi002 podman[77352]: 2024-02-13 15:17:52.710100596 +0000 UTC m=+1.178414095 container died eaf3a325b048edeba3b4d3809aa2c5924f8dc04d4510be2de1aa90ad9345a838 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8) 2024-02-13T15:17:53.305 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:53 smithi002 podman[77352]: 2024-02-13 15:17:53.223017424 +0000 UTC m=+1.691330977 container cleanup eaf3a325b048edeba3b4d3809aa2c5924f8dc04d4510be2de1aa90ad9345a838 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, GIT_CLEAN=True, io.buildah.version=1.19.8) 2024-02-13T15:17:53.305 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:53 smithi002 bash[77352]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: osdmap e120: 8 total, 7 up, 8 in 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:53 smithi098 ceph-mon[58041]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: osdmap e120: 8 total, 7 up, 8 in 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[64469]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: osdmap e120: 8 total, 7 up, 8 in 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.628 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:53 smithi002 ceph-mon[67023]: failed to encode map e120 with expected crc 2024-02-13T15:17:53.629 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:53 smithi002 podman[77369]: 2024-02-13 15:17:53.466181403 +0000 UTC m=+0.752470057 container remove eaf3a325b048edeba3b4d3809aa2c5924f8dc04d4510be2de1aa90ad9345a838 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public) 2024-02-13T15:17:54.212 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:53 smithi002 podman[77459]: 2024-02-13 15:17:53.79852398 +0000 UTC m=+0.017533286 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:54.212 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:53 smithi002 podman[77459]: 2024-02-13 15:17:53.908906567 +0000 UTC m=+0.127915904 container create 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-13T15:17:54.212 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:54 smithi002 podman[77459]: 2024-02-13 15:17:54.128351031 +0000 UTC m=+0.347360337 container init 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-13T15:17:54.212 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:54 smithi002 podman[77459]: 2024-02-13 15:17:54.131550974 +0000 UTC m=+0.350560275 container start 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:17:54.212 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:54 smithi002 podman[77459]: 2024-02-13 15:17:54.211464496 +0000 UTC m=+0.430473812 container attach 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: pgmap v112: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: Health check failed: Reduced data availability: 8 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: osdmap e121: 8 total, 7 up, 8 in 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:54 smithi098 ceph-mon[58041]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:54 smithi002 podman[77459]: 2024-02-13 15:17:54.438268556 +0000 UTC m=+0.657277857 container died 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-13T15:17:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: pgmap v112: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: Health check failed: Reduced data availability: 8 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:54.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: osdmap e121: 8 total, 7 up, 8 in 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[67023]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: pgmap v112: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:54.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: Health check failed: Reduced data availability: 8 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: osdmap e121: 8 total, 7 up, 8 in 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:54.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:54 smithi002 ceph-mon[64469]: failed to encode map e121 with expected crc 2024-02-13T15:17:55.300 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77485]: 2024-02-13 15:17:55.031672736 +0000 UTC m=+0.582504374 container remove 32146d97b5cdafa44a0c1fa1703d5d2bacc93d1a45fa76b1ae0424ba562d4a4e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS) 2024-02-13T15:17:55.300 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-13T15:17:55.301 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 systemd[1]: Stopped Ceph osd.3 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:17:55.301 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.3.service: Consumed 8.229s CPU time. 2024-02-13T15:17:55.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 systemd[1]: Starting Ceph osd.3 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:17:55.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77587]: 2024-02-13 15:17:55.417976295 +0000 UTC m=+0.018401853 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:55.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77587]: 2024-02-13 15:17:55.533064869 +0000 UTC m=+0.133490463 container create 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-13T15:17:55.903 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77587]: 2024-02-13 15:17:55.827837932 +0000 UTC m=+0.428263480 container init 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:17:55.903 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77587]: 2024-02-13 15:17:55.830940009 +0000 UTC m=+0.431365566 container start 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-13T15:17:56.280 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:55 smithi002 podman[77587]: 2024-02-13 15:17:55.902700105 +0000 UTC m=+0.503125653 container attach 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-13T15:17:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:56 smithi098 ceph-mon[58041]: pgmap v114: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-mon[64469]: pgmap v114: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:56.615 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-mon[67023]: pgmap v114: 161 pgs: 33 peering, 12 stale+active+clean, 116 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 bash[77587]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-13T15:17:56.616 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-13T15:17:56.617 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-13T15:17:56.617 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-13T15:17:56.617 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-13T15:17:56.617 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate[77603]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-13T15:17:56.617 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 podman[77587]: 2024-02-13 15:17:56.303285999 +0000 UTC m=+0.903711555 container died 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-13T15:17:57.316 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:56 smithi002 podman[77740]: 2024-02-13 15:17:56.940350027 +0000 UTC m=+0.631857966 container remove 22bf6a443574ef4f9f1782923b20281bb2dc039e1f70f8b76ca34c31502cbb16 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-13T15:17:57.316 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 podman[77806]: 2024-02-13 15:17:57.057918156 +0000 UTC m=+0.018530414 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:17:57.316 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 podman[77806]: 2024-02-13 15:17:57.174493308 +0000 UTC m=+0.135105556 container create e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:17:57.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:57 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 11 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:57.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:57 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 11 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:57.627 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 podman[77806]: 2024-02-13 15:17:57.410514076 +0000 UTC m=+0.371126328 container init e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=) 2024-02-13T15:17:57.627 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 podman[77806]: 2024-02-13 15:17:57.413693909 +0000 UTC m=+0.374306160 container start e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:17:57.627 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 ceph-osd[77826]: -- 172.21.15.2:0/2947271787 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55714a5d11e0 con 0x55714a5a3400 2024-02-13T15:17:57.627 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 bash[77806]: e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 2024-02-13T15:17:57.627 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 systemd[1]: Started Ceph osd.3 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:17:57.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:57 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 11 pgs degraded (PG_DEGRADED) 2024-02-13T15:17:58.376 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:17:57 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:17:57.961+0000 7f65497927c0 -1 Falling back to public interface 2024-02-13T15:17:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:58 smithi098 ceph-mon[58041]: pgmap v115: 161 pgs: 11 active+undersized+degraded, 33 peering, 22 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-02-13T15:17:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:58.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:17:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:17:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:17:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[64469]: pgmap v115: 161 pgs: 11 active+undersized+degraded, 33 peering, 22 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-02-13T15:17:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:58.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[67023]: pgmap v115: 161 pgs: 11 active+undersized+degraded, 33 peering, 22 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-02-13T15:17:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:58.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:17:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:17:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:17:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:17:59.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:17:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:00.129 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:17:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:17:59] "GET /metrics HTTP/1.1" 200 38480 "" "Prometheus/2.33.4" 2024-02-13T15:18:00.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:00 smithi098 ceph-mon[58041]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:00.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:00 smithi098 ceph-mon[58041]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 14 pgs peering) 2024-02-13T15:18:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-mon[64469]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-mon[64469]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 14 pgs peering) 2024-02-13T15:18:00.874 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-mon[67023]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:00.875 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-mon[67023]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 14 pgs peering) 2024-02-13T15:18:00.875 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:18:00.672+0000 7f65497927c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-13T15:18:00.875 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:18:00 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:18:00.698+0000 7f65497927c0 -1 osd.3 119 log_to_monitors true 2024-02-13T15:18:01.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:01 smithi098 ceph-mon[58041]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:18:01.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:01 smithi002 ceph-mon[64469]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:18:01.846 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:01 smithi002 ceph-mon[67023]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T15:18:02.126 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:18:01 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:18:01.847+0000 7f6536be2700 -1 osd.3 119 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: pgmap v117: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: failed to encode map e122 with expected crc 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: osdmap e122: 8 total, 7 up, 8 in 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: pgmap v117: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:18:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: failed to encode map e122 with expected crc 2024-02-13T15:18:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: osdmap e122: 8 total, 7 up, 8 in 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: pgmap v117: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T15:18:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: failed to encode map e122 with expected crc 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: osdmap e122: 8 total, 7 up, 8 in 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154]' entity='osd.3' 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:02.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e122 with expected crc 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154] boot 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: osdmap e123: 8 total, 8 up, 8 in 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:03 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e122 with expected crc 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154] boot 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: osdmap e123: 8 total, 8 up, 8 in 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e122 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: osd.3 [v2:172.21.15.2:6826/1505935154,v1:172.21.15.2:6827/1505935154] boot 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: osdmap e123: 8 total, 8 up, 8 in 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.856 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.856 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.856 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: failed to encode map e123 with expected crc 2024-02-13T15:18:03.856 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:03.856 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:03 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: pgmap v120: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: osdmap e124: 8 total, 8 up, 8 in 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:04 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: pgmap v120: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: osdmap e124: 8 total, 8 up, 8 in 2024-02-13T15:18:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: pgmap v120: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: osdmap e124: 8 total, 8 up, 8 in 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:04 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:05.511 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:18:05.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:05 smithi098 ceph-mon[58041]: failed to encode map e124 with expected crc 2024-02-13T15:18:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:05 smithi002 ceph-mon[64469]: failed to encode map e124 with expected crc 2024-02-13T15:18:05.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:05 smithi002 ceph-mon[67023]: failed to encode map e124 with expected crc 2024-02-13T15:18:06.224 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:18:06.224 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (5m) 4s ago 10m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:18:06.224 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (4m) 2m ago 9m 45.6M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (3m) 4s ago 8m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (4m) 4s ago 16m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (3m) 4s ago 16m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (2m) 2m ago 14m 20.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (3m) 4s ago 14m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (5m) 4s ago 10m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (4m) 2m ago 10m 17.0M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (114s) 4s ago 14m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (79s) 4s ago 13m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (44s) 4s ago 13m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (9s) 4s ago 12m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (12m) 2m ago 12m 53.6M 2992M 17.2.0 e1d6a67b021e a22874257222 2024-02-13T15:18:06.225 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (12m) 2m ago 12m 51.1M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (11m) 2m ago 11m 49.2M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (11m) 2m ago 11m 52.3M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (3m) 2m ago 10m 58.1M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (9m) 4s ago 9m 74.4M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (8m) 2m ago 8m 74.6M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (8m) 4s ago 8m 72.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:18:06.226 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (8m) 2m ago 8m 74.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:18:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 4 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 9 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:18:06.638 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:18:06.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:06 smithi098 ceph-mon[58041]: pgmap v122: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:06.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:06 smithi098 ceph-mon[58041]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:06 smithi002 ceph-mon[64469]: pgmap v122: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:06.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:06 smithi002 ceph-mon[64469]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:06.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:06 smithi002 ceph-mon[67023]: pgmap v122: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-02-13T15:18:06.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:06 smithi002 ceph-mon[67023]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "9/23 daemons upgraded", 2024-02-13T15:18:06.979 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:18:06.980 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:18:06.980 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:18:07.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:18:07.159+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-02-13T15:18:07.389 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.9 is active+undersized+degraded, acting [4,2] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-02-13T15:18:07.390 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3203765192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:07.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:07 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1425053389' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:07.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3203765192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1425053389' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:07.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3203765192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:07.878 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:07 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1425053389' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:08 smithi098 ceph-mon[58041]: pgmap v123: 161 pgs: 7 active+undersized+degraded, 11 active+undersized, 143 active+clean; 585 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 20/627 objects degraded (3.190%) 2024-02-13T15:18:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:08 smithi098 ceph-mon[58041]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:08 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:08 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-13T15:18:09.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:18:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:18:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:18:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[64469]: pgmap v123: 161 pgs: 7 active+undersized+degraded, 11 active+undersized, 143 active+clean; 585 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 20/627 objects degraded (3.190%) 2024-02-13T15:18:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[64469]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-13T15:18:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[67023]: pgmap v123: 161 pgs: 7 active+undersized+degraded, 11 active+undersized, 143 active+clean; 585 KiB data, 175 MiB used, 715 GiB / 715 GiB avail; 20/627 objects degraded (3.190%) 2024-02-13T15:18:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[67023]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:08 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-02-13T15:18:10.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:09 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:09 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:10.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:09 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:10.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:09] "GET /metrics HTTP/1.1" 200 38337 "" "Prometheus/2.33.4" 2024-02-13T15:18:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:10 smithi098 ceph-mon[58041]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:10 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 7 pgs degraded) 2024-02-13T15:18:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:10 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[64469]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 7 pgs degraded) 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[67023]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/627 objects degraded (3.190%), 7 pgs degraded) 2024-02-13T15:18:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:10 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:18:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:12 smithi098 ceph-mon[58041]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:12 smithi002 ceph-mon[64469]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:13.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:12 smithi002 ceph-mon[67023]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:14 smithi098 ceph-mon[58041]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:14 smithi002 ceph-mon[64469]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:15.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:14 smithi002 ceph-mon[67023]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:17.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:16 smithi098 ceph-mon[58041]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:16 smithi002 ceph-mon[64469]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:17.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:16 smithi002 ceph-mon[67023]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:18:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:17 smithi098 ceph-mon[58041]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:18:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[64469]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[67023]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T15:18:18.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:18.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:18.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:18 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:19.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:18:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:18:18.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-13T15:18:19.082 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:18:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:18:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"smithi002:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.98:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-13T15:18:19.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:18 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:19.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:18 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:20.294 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:19 smithi002 ceph-mon[64469]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:20.294 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:19] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.33.4" 2024-02-13T15:18:20.295 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:19 smithi002 ceph-mon[67023]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:19 smithi098 ceph-mon[58041]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:22.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:21 smithi098 ceph-mon[58041]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:21 smithi002 ceph-mon[64469]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:22.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:21 smithi002 ceph-mon[67023]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:23.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:22 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:22 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.042 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:22 smithi098 ceph-mon[58041]: Upgrade: osd.4 is safe to restart 2024-02-13T15:18:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[64469]: Upgrade: osd.4 is safe to restart 2024-02-13T15:18:23.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-13T15:18:23.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:22 smithi002 ceph-mon[67023]: Upgrade: osd.4 is safe to restart 2024-02-13T15:18:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:23 smithi098 ceph-mon[58041]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:23 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:23 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:18:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:23 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[64469]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:18:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[67023]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T15:18:24.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:23 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:25.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:24 smithi098 ceph-mon[58041]: Upgrade: Updating osd.4 2024-02-13T15:18:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:24 smithi098 ceph-mon[58041]: Deploying daemon osd.4 on smithi098 2024-02-13T15:18:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:24 smithi002 ceph-mon[64469]: Upgrade: Updating osd.4 2024-02-13T15:18:25.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:24 smithi002 ceph-mon[64469]: Deploying daemon osd.4 on smithi098 2024-02-13T15:18:25.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:24 smithi002 ceph-mon[67023]: Upgrade: Updating osd.4 2024-02-13T15:18:25.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:24 smithi002 ceph-mon[67023]: Deploying daemon osd.4 on smithi098 2024-02-13T15:18:26.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:25 smithi098 ceph-mon[58041]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:25 smithi002 ceph-mon[64469]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:25 smithi002 ceph-mon[67023]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:27.332 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:27 smithi098 systemd[1]: Stopping Ceph osd.4 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:18:27.832 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[29067]: 2024-02-13T15:18:27.357+0000 7f52da8f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:18:27.832 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[29067]: 2024-02-13T15:18:27.357+0000 7f52da8f9700 -1 osd.4 124 *** Got signal Terminated *** 2024-02-13T15:18:27.832 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:27 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[29067]: 2024-02-13T15:18:27.357+0000 7f52da8f9700 -1 osd.4 124 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:18:28.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:27 smithi098 ceph-mon[58041]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:28.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:27 smithi098 ceph-mon[58041]: osd.4 marked itself down and dead 2024-02-13T15:18:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:27 smithi002 ceph-mon[64469]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:27 smithi002 ceph-mon[64469]: osd.4 marked itself down and dead 2024-02-13T15:18:28.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:27 smithi002 ceph-mon[67023]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:28.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:27 smithi002 ceph-mon[67023]: osd.4 marked itself down and dead 2024-02-13T15:18:28.753 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:28 smithi098 podman[60859]: 2024-02-13 15:18:28.462887671 +0000 UTC m=+1.233990149 container died a228742572224c90330ac934e8f17dbcd59669d1fd975b57b2556e1167b21f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, version=8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-13T15:18:29.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:28 smithi098 podman[60859]: 2024-02-13 15:18:28.867426553 +0000 UTC m=+1.638529036 container cleanup a228742572224c90330ac934e8f17dbcd59669d1fd975b57b2556e1167b21f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, release=754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_CLEAN=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-13T15:18:29.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:28 smithi098 bash[60859]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: osdmap e125: 8 total, 7 up, 8 in 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:28 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:18:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: osdmap e125: 8 total, 7 up, 8 in 2024-02-13T15:18:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: osdmap e125: 8 total, 7 up, 8 in 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: failed to encode map e125 with expected crc 2024-02-13T15:18:29.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:28 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:29.437 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60877]: 2024-02-13 15:18:29.119116997 +0000 UTC m=+0.645040653 container remove a228742572224c90330ac934e8f17dbcd59669d1fd975b57b2556e1167b21f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, GIT_BRANCH=HEAD, release=754, version=8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vcs-type=git, name=centos-stream, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-13T15:18:29.765 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.436794379 +0000 UTC m=+0.101039062 container create 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-13T15:18:29.765 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.353731296 +0000 UTC m=+0.017975946 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:18:29.765 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.673172349 +0000 UTC m=+0.337417003 container init 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-13T15:18:29.765 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.676884518 +0000 UTC m=+0.341129168 container start 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:18:30.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.764444293 +0000 UTC m=+0.428688968 container attach 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-13T15:18:30.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:29 smithi098 podman[60966]: 2024-02-13 15:18:29.96819403 +0000 UTC m=+0.632438666 container died 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0) 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: osdmap e126: 8 total, 7 up, 8 in 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:29 smithi098 ceph-mon[58041]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:29] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.33.4" 2024-02-13T15:18:30.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:30.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: osdmap e126: 8 total, 7 up, 8 in 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[67023]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: osdmap e126: 8 total, 7 up, 8 in 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:30.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:29 smithi002 ceph-mon[64469]: failed to encode map e126 with expected crc 2024-02-13T15:18:31.028 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:30 smithi098 podman[60993]: 2024-02-13 15:18:30.593788644 +0000 UTC m=+0.616405336 container remove 8ec0c944815aafaf96507d259d3bf3ad1b7790393626bd7058e309dac912831d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-13T15:18:31.029 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:30 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-13T15:18:31.029 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:30 smithi098 systemd[1]: Stopped Ceph osd.4 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:18:31.029 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:30 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.4.service: Consumed 8.875s CPU time. 2024-02-13T15:18:31.029 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:30 smithi098 systemd[1]: Starting Ceph osd.4 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:18:31.281 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:31.027986364 +0000 UTC m=+0.112967313 container create 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-13T15:18:31.281 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:30.933474523 +0000 UTC m=+0.018455486 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:18:31.281 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:31.280634533 +0000 UTC m=+0.365615492 container init 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:18:31.582 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:31.28373809 +0000 UTC m=+0.368719046 container start 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:18:31.582 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:31.355730475 +0000 UTC m=+0.440711429 container attach 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-02-13T15:18:32.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-mon[58041]: pgmap v137: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:18:32.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:32.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:32.044 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-13T15:18:32.044 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 bash[61099]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate[61114]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-13T15:18:32.045 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:31 smithi098 podman[61099]: 2024-02-13 15:18:31.749898828 +0000 UTC m=+0.834879779 container died 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True) 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[64469]: pgmap v137: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[67023]: pgmap v137: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:32.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:31 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:32.646 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 podman[61247]: 2024-02-13 15:18:32.376964257 +0000 UTC m=+0.618720962 container remove 0c23dfe0462cbd376e62dc60e8aabed5859697f95e7bb7676c345662d4bbf692 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-13T15:18:32.647 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 podman[61312]: 2024-02-13 15:18:32.578016436 +0000 UTC m=+0.101465399 container create 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:18:32.647 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 podman[61312]: 2024-02-13 15:18:32.494118764 +0000 UTC m=+0.017567673 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:18:33.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 podman[61312]: 2024-02-13 15:18:32.755355677 +0000 UTC m=+0.278804584 container init 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-13T15:18:33.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 podman[61312]: 2024-02-13 15:18:32.758363068 +0000 UTC m=+0.281811970 container start 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:18:33.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 ceph-osd[61331]: -- 172.21.15.98:0/396226394 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fea19b1e0 con 0x558fea16d400 2024-02-13T15:18:33.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 bash[61312]: 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 2024-02-13T15:18:33.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:32 smithi098 systemd[1]: Started Ceph osd.4 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:18:33.582 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:33 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:18:33.301+0000 7f78281a77c0 -1 Falling back to public interface 2024-02-13T15:18:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:33 smithi098 ceph-mon[58041]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:33 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:33 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:33 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[64469]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[67023]: pgmap v138: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:33 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded (PG_DEGRADED) 2024-02-13T15:18:36.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:35 smithi098 ceph-mon[58041]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:36.264 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:18:36.004+0000 7f78281a77c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-13T15:18:36.264 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:18:36.033+0000 7f78281a77c0 -1 osd.4 124 log_to_monitors true 2024-02-13T15:18:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:35 smithi002 ceph-mon[64469]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:36.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:35 smithi002 ceph-mon[67023]: pgmap v139: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:37.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:18:36 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:18:36.926+0000 7f78195ff700 -1 osd.4 124 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:18:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:36 smithi098 ceph-mon[58041]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:36 smithi098 ceph-mon[58041]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:36 smithi002 ceph-mon[64469]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:36 smithi002 ceph-mon[64469]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:36 smithi002 ceph-mon[67023]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:36 smithi002 ceph-mon[67023]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T15:18:37.760 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:18:38.025 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: pgmap v140: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: osdmap e127: 8 total, 7 up, 8 in 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[64469]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:38.026 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: pgmap v140: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: osdmap e127: 8 total, 7 up, 8 in 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.027 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:37 smithi002 ceph-mon[67023]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: pgmap v140: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: osdmap e127: 8 total, 7 up, 8 in 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: failed to encode map e127 with expected crc 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:37 smithi098 ceph-mon[58041]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:38.462 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:18:38.462 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (5m) 36s ago 10m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:18:38.462 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (4m) 1s ago 10m 54.8M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (4m) 36s ago 9m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (4m) 1s ago 14m 450M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (4m) 36s ago 16m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (4m) 36s ago 16m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (3m) 1s ago 15m 37.9M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (3m) 36s ago 15m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (5m) 36s ago 11m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (5m) 1s ago 11m 17.5M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (2m) 36s ago 14m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (111s) 36s ago 14m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (76s) 36s ago 13m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:18:38.463 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (41s) 36s ago 13m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (5s) 1s ago 13m 12.6M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (12m) 1s ago 12m 56.4M 2992M 17.2.0 e1d6a67b021e 5e9b3ab2b041 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (12m) 1s ago 12m 55.4M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (11m) 1s ago 11m 58.9M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (4m) 1s ago 11m 69.7M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (9m) 36s ago 9m 74.4M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (9m) 1s ago 9m 75.5M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (9m) 36s ago 9m 72.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:18:38.464 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (9m) 1s ago 9m 75.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:18:38.867 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 5 2024-02-13T15:18:38.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 10 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:18:38.869 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:18:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: failed to encode map e127 with expected crc 2024-02-13T15:18:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538] boot 2024-02-13T15:18:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: osdmap e128: 8 total, 8 up, 8 in 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2082236108' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[67023]: osdmap e129: 8 total, 8 up, 8 in 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: failed to encode map e127 with expected crc 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:39.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538] boot 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: osdmap e128: 8 total, 8 up, 8 in 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2082236108' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:39.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:38 smithi002 ceph-mon[64469]: osdmap e129: 8 total, 8 up, 8 in 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: failed to encode map e127 with expected crc 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: osd.4 [v2:172.21.15.98:6800/848667538,v1:172.21.15.98:6801/848667538] boot 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: osdmap e128: 8 total, 8 up, 8 in 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: failed to encode map e128 with expected crc 2024-02-13T15:18:39.206 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2082236108' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:39.207 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:38 smithi098 ceph-mon[58041]: osdmap e129: 8 total, 8 up, 8 in 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:18:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:18:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "10/23 daemons upgraded", 2024-02-13T15:18:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:18:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:18:39.227 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-02-13T15:18:39.633 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.9 is active+undersized+degraded, acting [2,3] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-02-13T15:18:39.634 INFO:teuthology.orchestra.run.smithi002.stdout: pg 5.1d is active+undersized+degraded, acting [1,0] 2024-02-13T15:18:39.635 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-02-13T15:18:39.923 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:39] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.33.4" 2024-02-13T15:18:40.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: pgmap v143: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:40.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:39 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/644645322' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: pgmap v143: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/644645322' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: pgmap v143: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: failed to encode map e129 with expected crc 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:18:40.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:39 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/644645322' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: pgmap v145: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 105/627 objects degraded (16.746%) 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:41 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:18:42.012+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: pgmap v145: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 105/627 objects degraded (16.746%) 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: pgmap v145: 161 pgs: 26 active+undersized+degraded, 42 active+undersized, 93 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 105/627 objects degraded (16.746%) 2024-02-13T15:18:42.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:42.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:42.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:18:42.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:41 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:42 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:43.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:42 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-13T15:18:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:43 smithi098 ceph-mon[58041]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:43 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded) 2024-02-13T15:18:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:43 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:18:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[64469]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded) 2024-02-13T15:18:44.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:18:44.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[67023]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/627 objects degraded (16.746%), 26 pgs degraded) 2024-02-13T15:18:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:43 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:18:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:45 smithi098 ceph-mon[58041]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T15:18:46.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:45 smithi002 ceph-mon[64469]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T15:18:46.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:45 smithi002 ceph-mon[67023]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T15:18:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:47 smithi098 ceph-mon[58041]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:18:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[64469]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[67023]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:48.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:18:49.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:48 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:49.277 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:48 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:49.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:48 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:50.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:49 smithi098 ceph-mon[58041]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:50.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:49 smithi002 ceph-mon[64469]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:50.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:49] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-02-13T15:18:50.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:49 smithi002 ceph-mon[67023]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:18:52.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:51 smithi098 ceph-mon[58041]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-13T15:18:52.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:51 smithi002 ceph-mon[64469]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-13T15:18:52.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:51 smithi002 ceph-mon[67023]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-13T15:18:53.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:53 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:53.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:53.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:53 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:54.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:54 smithi098 ceph-mon[58041]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:54.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:54 smithi002 ceph-mon[64469]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:54.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:54 smithi002 ceph-mon[67023]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:56.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:56 smithi098 ceph-mon[58041]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:56.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:56 smithi002 ceph-mon[64469]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:56.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:56 smithi002 ceph-mon[67023]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:18:58.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:58 smithi098 ceph-mon[58041]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:58.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:58 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:58 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.250 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:58 smithi098 ceph-mon[58041]: Upgrade: osd.5 is safe to restart 2024-02-13T15:18:58.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[64469]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:58.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[64469]: Upgrade: osd.5 is safe to restart 2024-02-13T15:18:58.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[67023]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:18:58.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-13T15:18:58.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:58 smithi002 ceph-mon[67023]: Upgrade: osd.5 is safe to restart 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: Upgrade: Updating osd.5 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:18:59 smithi098 ceph-mon[58041]: Deploying daemon osd.5 on smithi098 2024-02-13T15:18:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: Upgrade: Updating osd.5 2024-02-13T15:18:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:18:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[64469]: Deploying daemon osd.5 on smithi098 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: Upgrade: Updating osd.5 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:18:59.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-mon[67023]: Deploying daemon osd.5 on smithi098 2024-02-13T15:19:00.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:18:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:18:59] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-02-13T15:19:00.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:00 smithi098 ceph-mon[58041]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:00 smithi002 ceph-mon[64469]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:00.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:00 smithi002 ceph-mon[67023]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:02.082 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:01 smithi098 systemd[1]: Stopping Ceph osd.5 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:19:02.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[32007]: 2024-02-13T15:19:02.085+0000 7f3f14dc5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:19:02.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[32007]: 2024-02-13T15:19:02.085+0000 7f3f14dc5700 -1 osd.5 129 *** Got signal Terminated *** 2024-02-13T15:19:02.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[32007]: 2024-02-13T15:19:02.085+0000 7f3f14dc5700 -1 osd.5 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:19:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[64469]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:02.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[64469]: osd.5 marked itself down and dead 2024-02-13T15:19:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[67023]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:02.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:02 smithi002 ceph-mon[67023]: osd.5 marked itself down and dead 2024-02-13T15:19:03.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-mon[58041]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:03.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:03.038 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:02 smithi098 ceph-mon[58041]: osd.5 marked itself down and dead 2024-02-13T15:19:03.332 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:03 smithi098 podman[63206]: 2024-02-13 15:19:03.036859482 +0000 UTC m=+1.076367210 container died 5e9b3ab2b0418e0b14536f909011f85567ed89514010e74a8d9a280b99fdc8a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, ceph=True, build-date=2022-05-03T08:36:31.336870) 2024-02-13T15:19:03.629 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:03 smithi098 podman[63206]: 2024-02-13 15:19:03.436621522 +0000 UTC m=+1.476129254 container cleanup 5e9b3ab2b0418e0b14536f909011f85567ed89514010e74a8d9a280b99fdc8a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, version=8, RELEASE=HEAD, architecture=x86_64, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-02-13T15:19:03.629 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:03 smithi098 bash[63206]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5 2024-02-13T15:19:04.009 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:03 smithi098 podman[63223]: 2024-02-13 15:19:03.679914841 +0000 UTC m=+0.631964353 container remove 5e9b3ab2b0418e0b14536f909011f85567ed89514010e74a8d9a280b99fdc8a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, com.redhat.component=centos-stream-container, release=754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, architecture=x86_64, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8) 2024-02-13T15:19:04.009 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:04.009 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: osdmap e130: 8 total, 7 up, 8 in 2024-02-13T15:19:04.009 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.009 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.010 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.010 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.010 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.010 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:03 smithi098 ceph-mon[58041]: osdmap e131: 8 total, 7 up, 8 in 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: osdmap e130: 8 total, 7 up, 8 in 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[64469]: osdmap e131: 8 total, 7 up, 8 in 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: osdmap e130: 8 total, 7 up, 8 in 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: failed to encode map e130 with expected crc 2024-02-13T15:19:04.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:03 smithi002 ceph-mon[67023]: osdmap e131: 8 total, 7 up, 8 in 2024-02-13T15:19:04.293 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:03.912335891 +0000 UTC m=+0.018641506 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:04.293 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:04.047512495 +0000 UTC m=+0.153818093 container create e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-13T15:19:04.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:04.29240857 +0000 UTC m=+0.398714173 container init e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-13T15:19:04.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:04.295727491 +0000 UTC m=+0.402033090 container start e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-13T15:19:04.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:04.409254483 +0000 UTC m=+0.515560097 container attach e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:19:04.880 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:04 smithi098 podman[63313]: 2024-02-13 15:19:04.591803086 +0000 UTC m=+0.698108685 container died e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: pgmap v157: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: failed to encode map e131 with expected crc 2024-02-13T15:19:04.881 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:04 smithi098 ceph-mon[58041]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-13T15:19:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: pgmap v157: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:19:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[64469]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: pgmap v157: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: failed to encode map e131 with expected crc 2024-02-13T15:19:05.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:04 smithi002 ceph-mon[67023]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-13T15:19:05.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 podman[63339]: 2024-02-13 15:19:05.207590725 +0000 UTC m=+0.606873305 container remove e54c82f97d531686ab00c49abdac0f5b6afc5bef475e8b1bc81120661768c359 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:19:05.583 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-13T15:19:05.583 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 systemd[1]: Stopped Ceph osd.5 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:19:05.583 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.5.service: Consumed 8.981s CPU time. 2024-02-13T15:19:05.583 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 systemd[1]: Starting Ceph osd.5 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:19:05.985 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 podman[63443]: 2024-02-13 15:19:05.576564457 +0000 UTC m=+0.019841296 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:05.986 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 podman[63443]: 2024-02-13 15:19:05.724068869 +0000 UTC m=+0.167345696 container create 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-13T15:19:06.332 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 podman[63443]: 2024-02-13 15:19:05.984969186 +0000 UTC m=+0.428246022 container init 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-13T15:19:06.332 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:05 smithi098 podman[63443]: 2024-02-13 15:19:05.988312662 +0000 UTC m=+0.431589490 container start 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-13T15:19:06.332 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 podman[63443]: 2024-02-13 15:19:06.09296907 +0000 UTC m=+0.536245901 container attach 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , ceph=True) 2024-02-13T15:19:06.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-mon[58041]: pgmap v159: 161 pgs: 16 stale+active+clean, 15 peering, 130 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:19:06.832 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 bash[63443]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-13T15:19:06.833 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-13T15:19:06.834 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-13T15:19:06.834 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T15:19:06.834 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate[63457]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-13T15:19:06.834 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:06 smithi098 podman[63443]: 2024-02-13 15:19:06.449995993 +0000 UTC m=+0.893272819 container died 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-13T15:19:07.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:06 smithi002 ceph-mon[64469]: pgmap v159: 161 pgs: 16 stale+active+clean, 15 peering, 130 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:19:07.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:06 smithi002 ceph-mon[67023]: pgmap v159: 161 pgs: 16 stale+active+clean, 15 peering, 130 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-13T15:19:07.431 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 podman[63575]: 2024-02-13 15:19:07.180238829 +0000 UTC m=+0.721845227 container remove 8fd34aeea52380ecb88b4e3c54450f397b4b5d7f682235b0c7af242c070d0c5d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-activate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-13T15:19:07.431 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 podman[63639]: 2024-02-13 15:19:07.304542184 +0000 UTC m=+0.018952524 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:07.432 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 podman[63639]: 2024-02-13 15:19:07.431188547 +0000 UTC m=+0.145598875 container create edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-13T15:19:07.684 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 podman[63639]: 2024-02-13 15:19:07.650767928 +0000 UTC m=+0.365178255 container init edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-13T15:19:07.684 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 podman[63639]: 2024-02-13 15:19:07.653877514 +0000 UTC m=+0.368287841 container start edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-13T15:19:07.684 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 ceph-osd[63658]: -- 172.21.15.98:0/2287996168 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555ea46e91e0 con 0x555ea46bb400 2024-02-13T15:19:08.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:07 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 42/627 objects degraded (6.699%), 12 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:08.052 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:07 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 42/627 objects degraded (6.699%), 12 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:08.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:07 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 42/627 objects degraded (6.699%), 12 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:08.082 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 bash[63639]: edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 2024-02-13T15:19:08.082 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:07 smithi098 systemd[1]: Started Ceph osd.5 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:19:08.582 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:19:08.196+0000 7fbba62227c0 -1 Falling back to public interface 2024-02-13T15:19:08.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:08 smithi098 ceph-mon[58041]: pgmap v160: 161 pgs: 4 stale+active+clean, 12 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 43 op/s; 42/627 objects degraded (6.699%) 2024-02-13T15:19:08.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:08 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:08.998 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:08 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[64469]: pgmap v160: 161 pgs: 4 stale+active+clean, 12 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 43 op/s; 42/627 objects degraded (6.699%) 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[67023]: pgmap v160: 161 pgs: 4 stale+active+clean, 12 active+undersized+degraded, 15 peering, 23 active+undersized, 107 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 43 op/s; 42/627 objects degraded (6.699%) 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:08 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:10.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:09 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:10.003 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:09] "GET /metrics HTTP/1.1" 200 38501 "" "Prometheus/2.33.4" 2024-02-13T15:19:10.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:09 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:10.003 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:19:10.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:09 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (6m) 68s ago 11m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (5m) 33s ago 10m 54.8M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (4m) 68s ago 9m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (4m) 33s ago 15m 450M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (5m) 68s ago 17m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (4m) 68s ago 17m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (3m) 33s ago 15m 37.9M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (4m) 68s ago 15m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (6m) 68s ago 11m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (5m) 33s ago 11m 17.5M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (2m) 68s ago 15m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (2m) 68s ago 14m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (108s) 68s ago 14m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (73s) 68s ago 13m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:19:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (38s) 33s ago 13m 12.6M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 starting - - - 2992M 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (12m) 33s ago 12m 55.4M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (12m) 33s ago 12m 58.9M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (4m) 33s ago 11m 69.7M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (10m) 68s ago 10m 74.4M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (10m) 33s ago 10m 75.5M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (10m) 68s ago 9m 72.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:19:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (9m) 33s ago 9m 75.5M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:19:11.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:10 smithi098 ceph-mon[58041]: pgmap v161: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 99 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:10 smithi098 ceph-mon[58041]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-02-13T15:19:11.082 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:19:10.889+0000 7fbba62227c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-13T15:19:11.082 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:19:10.916+0000 7fbba62227c0 -1 osd.5 129 log_to_monitors true 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 5 2024-02-13T15:19:11.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 10 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:19:11.100 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:19:11.100 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:10 smithi002 ceph-mon[64469]: pgmap v161: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 99 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:11.100 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:10 smithi002 ceph-mon[64469]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-02-13T15:19:11.101 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:10 smithi002 ceph-mon[67023]: pgmap v161: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 99 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:11.101 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:10 smithi002 ceph-mon[67023]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "10/23 daemons upgraded", 2024-02-13T15:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:19:11.464 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:19:11.464 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 74/627 objects degraded (11.802%), 21 pgs degraded 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: osd.5 (root=default,host=smithi098) is down 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 74/627 objects degraded (11.802%), 21 pgs degraded 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-02-13T15:19:11.874 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-02-13T15:19:11.875 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1561366590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:11.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:11.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:11 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1561366590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:12.029 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:12.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:12.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:12.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:12.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T15:19:12.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:11 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1561366590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:12.331 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:19:12.152+0000 7fbb93672700 -1 osd.5 129 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: pgmap v162: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 95 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='client.54341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: failed to encode map e132 with expected crc 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: osdmap e132: 8 total, 7 up, 8 in 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2616124830' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='osd.5 ' entity='osd.5' 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:12 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: pgmap v162: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 95 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='client.54341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: failed to encode map e132 with expected crc 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: osdmap e132: 8 total, 7 up, 8 in 2024-02-13T15:19:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2616124830' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='osd.5 ' entity='osd.5' 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: pgmap v162: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 95 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='client.54341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: failed to encode map e132 with expected crc 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: osdmap e132: 8 total, 7 up, 8 in 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2616124830' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:13.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='osd.5 ' entity='osd.5' 2024-02-13T15:19:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:13.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:12 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: failed to encode map e132 with expected crc 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683] boot 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: osdmap e133: 8 total, 8 up, 8 in 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:13 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: failed to encode map e132 with expected crc 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683] boot 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: osdmap e133: 8 total, 8 up, 8 in 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: failed to encode map e132 with expected crc 2024-02-13T15:19:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: osd.5 [v2:172.21.15.98:6808/3810816683,v1:172.21.15.98:6809/3810816683] boot 2024-02-13T15:19:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: osdmap e133: 8 total, 8 up, 8 in 2024-02-13T15:19:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T15:19:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:13 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:14.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: pgmap v164: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:14.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:14.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:14.703 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:14 smithi098 ceph-mon[58041]: osdmap e134: 8 total, 8 up, 8 in 2024-02-13T15:19:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: pgmap v164: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[64469]: osdmap e134: 8 total, 8 up, 8 in 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: pgmap v164: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: failed to encode map e133 with expected crc 2024-02-13T15:19:15.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:14 smithi002 ceph-mon[67023]: osdmap e134: 8 total, 8 up, 8 in 2024-02-13T15:19:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:15 smithi098 ceph-mon[58041]: failed to encode map e134 with expected crc 2024-02-13T15:19:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:15 smithi098 ceph-mon[58041]: failed to encode map e134 with expected crc 2024-02-13T15:19:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:15 smithi098 ceph-mon[58041]: failed to encode map e134 with expected crc 2024-02-13T15:19:15.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:15 smithi098 ceph-mon[58041]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[64469]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[64469]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[64469]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[64469]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[67023]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[67023]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[67023]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:15 smithi002 ceph-mon[67023]: failed to encode map e134 with expected crc 2024-02-13T15:19:16.981 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:16 smithi098 ceph-mon[58041]: pgmap v167: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:16.982 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:16 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 21 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:16 smithi002 ceph-mon[64469]: pgmap v167: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:16 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 21 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:17.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:16 smithi002 ceph-mon[67023]: pgmap v167: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s; 74/627 objects degraded (11.802%) 2024-02-13T15:19:17.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:16 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 21 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:17.626 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:19:17.213+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: pgmap v168: 161 pgs: 5 active+undersized+degraded, 10 active+undersized, 146 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 294 op/s; 14/627 objects degraded (2.233%); 0 B/s, 6 objects/s recovering 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:17 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-13T15:19:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: pgmap v168: 161 pgs: 5 active+undersized+degraded, 10 active+undersized, 146 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 294 op/s; 14/627 objects degraded (2.233%); 0 B/s, 6 objects/s recovering 2024-02-13T15:19:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:18.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-13T15:19:18.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: pgmap v168: 161 pgs: 5 active+undersized+degraded, 10 active+undersized, 146 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 294 op/s; 14/627 objects degraded (2.233%); 0 B/s, 6 objects/s recovering 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:18.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:17 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-13T15:19:19.280 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:18 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:19.280 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:18 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:19.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:18 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:20.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:19] "GET /metrics HTTP/1.1" 200 38512 "" "Prometheus/2.33.4" 2024-02-13T15:19:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[64469]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 201 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/627 objects degraded (2.233%), 5 pgs degraded) 2024-02-13T15:19:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:19:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[67023]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 201 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/627 objects degraded (2.233%), 5 pgs degraded) 2024-02-13T15:19:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:19 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:19:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:19 smithi098 ceph-mon[58041]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 201 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:19 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/627 objects degraded (2.233%), 5 pgs degraded) 2024-02-13T15:19:20.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:19 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:19:22.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:21 smithi098 ceph-mon[58041]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:22.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:21 smithi002 ceph-mon[64469]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:22.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:21 smithi002 ceph-mon[67023]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s; 0 B/s, 6 objects/s recovering 2024-02-13T15:19:24.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:23 smithi098 ceph-mon[58041]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s; 0 B/s, 5 objects/s recovering 2024-02-13T15:19:24.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:23 smithi002 ceph-mon[64469]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s; 0 B/s, 5 objects/s recovering 2024-02-13T15:19:24.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:23 smithi002 ceph-mon[67023]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s; 0 B/s, 5 objects/s recovering 2024-02-13T15:19:26.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:25 smithi098 ceph-mon[58041]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:26.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:25 smithi002 ceph-mon[64469]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:26.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:25 smithi002 ceph-mon[67023]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:28.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:28 smithi098 ceph-mon[58041]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:28.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:28 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:28 smithi002 ceph-mon[64469]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:28.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:28 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:28.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:28 smithi002 ceph-mon[67023]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 0 B/s, 4 objects/s recovering 2024-02-13T15:19:28.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:28 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:29.282 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:29 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:29.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:29 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:29.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:29 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:30 smithi002 ceph-mon[64469]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T15:19:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:29] "GET /metrics HTTP/1.1" 200 38512 "" "Prometheus/2.33.4" 2024-02-13T15:19:30.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:30 smithi002 ceph-mon[67023]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T15:19:30.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:30 smithi098 ceph-mon[58041]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T15:19:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:32 smithi098 ceph-mon[58041]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:32.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[64469]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:32.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[67023]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:32.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:33 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:33 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:33 smithi098 ceph-mon[58041]: Upgrade: osd.6 is safe to restart 2024-02-13T15:19:33.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[64469]: Upgrade: osd.6 is safe to restart 2024-02-13T15:19:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-13T15:19:33.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:33 smithi002 ceph-mon[67023]: Upgrade: osd.6 is safe to restart 2024-02-13T15:19:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:34 smithi098 ceph-mon[58041]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:34 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:34 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:19:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:34 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[64469]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[67023]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T15:19:34.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:34 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:35 smithi098 ceph-mon[58041]: Upgrade: Updating osd.6 2024-02-13T15:19:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:35 smithi098 ceph-mon[58041]: Deploying daemon osd.6 on smithi098 2024-02-13T15:19:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:35 smithi002 ceph-mon[64469]: Upgrade: Updating osd.6 2024-02-13T15:19:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:35 smithi002 ceph-mon[64469]: Deploying daemon osd.6 on smithi098 2024-02-13T15:19:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:35 smithi002 ceph-mon[67023]: Upgrade: Updating osd.6 2024-02-13T15:19:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:35 smithi002 ceph-mon[67023]: Deploying daemon osd.6 on smithi098 2024-02-13T15:19:36.322 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:36 smithi098 ceph-mon[58041]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:36 smithi002 ceph-mon[64469]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:36.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:36 smithi002 ceph-mon[67023]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:19:37.236 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:36 smithi098 systemd[1]: Stopping Ceph osd.6 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:19:37.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:37 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[34899]: 2024-02-13T15:19:37.233+0000 7f8c871bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:19:37.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:37 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[34899]: 2024-02-13T15:19:37.233+0000 7f8c871bc700 -1 osd.6 134 *** Got signal Terminated *** 2024-02-13T15:19:37.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:37 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[34899]: 2024-02-13T15:19:37.233+0000 7f8c871bc700 -1 osd.6 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:19:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:38 smithi098 ceph-mon[58041]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:38 smithi098 ceph-mon[58041]: osd.6 marked itself down and dead 2024-02-13T15:19:38.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:38 smithi002 ceph-mon[64469]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:38.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:38 smithi002 ceph-mon[64469]: osd.6 marked itself down and dead 2024-02-13T15:19:38.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:38 smithi002 ceph-mon[67023]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:19:38.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:38 smithi002 ceph-mon[67023]: osd.6 marked itself down and dead 2024-02-13T15:19:38.761 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:38 smithi098 podman[65531]: 2024-02-13 15:19:38.501779287 +0000 UTC m=+1.392516343 container died 2efee4dad73ecbac4186b75e42c3485fc39f61d9c97d314ba1ac9400398aa8d0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.19.8, version=8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, release=754, vendor=Red Hat, Inc., distribution-scope=public, ceph=True) 2024-02-13T15:19:39.036 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:38 smithi098 podman[65531]: 2024-02-13 15:19:38.858809383 +0000 UTC m=+1.749546444 container cleanup 2efee4dad73ecbac4186b75e42c3485fc39f61d9c97d314ba1ac9400398aa8d0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, GIT_CLEAN=True, release=754, io.openshift.tags=base centos centos-stream, architecture=x86_64, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public) 2024-02-13T15:19:39.036 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:38 smithi098 bash[65531]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: osdmap e135: 8 total, 7 up, 8 in 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:39 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:39.332 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65549]: 2024-02-13 15:19:39.168872616 +0000 UTC m=+0.660670696 container remove 2efee4dad73ecbac4186b75e42c3485fc39f61d9c97d314ba1ac9400398aa8d0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, io.buildah.version=1.19.8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: osdmap e135: 8 total, 7 up, 8 in 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: osdmap e135: 8 total, 7 up, 8 in 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: failed to encode map e135 with expected crc 2024-02-13T15:19:39.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:39.808 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65641]: 2024-02-13 15:19:39.519970277 +0000 UTC m=+0.109941625 container create 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-13T15:19:39.809 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65641]: 2024-02-13 15:19:39.428040698 +0000 UTC m=+0.018012052 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:39.809 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65641]: 2024-02-13 15:19:39.733015367 +0000 UTC m=+0.322986713 container init 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-13T15:19:39.809 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65641]: 2024-02-13 15:19:39.736115148 +0000 UTC m=+0.326086492 container start 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True) 2024-02-13T15:19:39.809 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:39 smithi098 podman[65641]: 2024-02-13 15:19:39.80802566 +0000 UTC m=+0.397996999 container attach 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-13T15:19:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:40 smithi098 ceph-mon[58041]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:19:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:40 smithi098 ceph-mon[58041]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:40 smithi098 ceph-mon[58041]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:40 smithi098 ceph-mon[58041]: osdmap e136: 8 total, 7 up, 8 in 2024-02-13T15:19:40.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 podman[65641]: 2024-02-13 15:19:40.026290478 +0000 UTC m=+0.616261808 container died 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-13T15:19:40.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:39] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2024-02-13T15:19:40.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[64469]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:19:40.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[64469]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[64469]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[64469]: osdmap e136: 8 total, 7 up, 8 in 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[67023]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[67023]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[67023]: failed to encode map e136 with expected crc 2024-02-13T15:19:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:40 smithi002 ceph-mon[67023]: osdmap e136: 8 total, 7 up, 8 in 2024-02-13T15:19:40.670 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 podman[65667]: 2024-02-13 15:19:40.637242676 +0000 UTC m=+0.605982503 container remove 397077a8a0e6a93a906d38861e9adaad89cec06f51b463c637bbc390e382270c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-13T15:19:40.671 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-13T15:19:40.671 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 systemd[1]: Stopped Ceph osd.6 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:19:40.671 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.6.service: Consumed 8.662s CPU time. 2024-02-13T15:19:41.077 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:40 smithi098 systemd[1]: Starting Ceph osd.6 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:19:41.332 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:40.97929022 +0000 UTC m=+0.017775096 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:41.332 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:41.139028856 +0000 UTC m=+0.177513731 container create b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-13T15:19:41.805 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:41.366873134 +0000 UTC m=+0.405358009 container init b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-13T15:19:41.805 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:41.370219268 +0000 UTC m=+0.408704155 container start b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-13T15:19:41.805 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:41.441825358 +0000 UTC m=+0.480310248 container attach b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-13T15:19:42.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:42 smithi098 ceph-mon[58041]: pgmap v182: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:19:42.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-13T15:19:42.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-13T15:19:42.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-13T15:19:42.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-13T15:19:42.082 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 bash[65772]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate[65786]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-13T15:19:42.083 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:41 smithi098 podman[65772]: 2024-02-13 15:19:41.828926482 +0000 UTC m=+0.867411352 container died b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-13T15:19:42.240 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:19:42.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:42 smithi002 ceph-mon[64469]: pgmap v182: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:19:42.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:42 smithi002 ceph-mon[67023]: pgmap v182: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:19:42.721 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 podman[65904]: 2024-02-13 15:19:42.39597061 +0000 UTC m=+0.557567232 container remove b64d46e5100edcd39596ba49f6570f93b4c3383f61f31d361ca9f8ffbcb497a3 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T15:19:42.722 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 podman[65965]: 2024-02-13 15:19:42.514794837 +0000 UTC m=+0.017917884 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:19:42.722 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 podman[65965]: 2024-02-13 15:19:42.613608903 +0000 UTC m=+0.116731989 container create 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-13T15:19:42.909 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (6m) 100s ago 11m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (6m) 30s ago 11m 55.2M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (5m) 100s ago 10m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (5m) 30s ago 16m 450M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (5m) 100s ago 17m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (5m) 100s ago 17m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (4m) 30s ago 16m 39.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (4m) 100s ago 16m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (6m) 100s ago 12m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:19:42.910 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (6m) 30s ago 12m 21.3M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (3m) 100s ago 15m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (2m) 100s ago 15m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (2m) 100s ago 14m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (105s) 100s ago 14m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (70s) 30s ago 14m 44.9M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (35s) 30s ago 13m 13.6M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed edf5d8026920 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (13m) 30s ago 13m 57.8M 2992M 17.2.0 e1d6a67b021e 2efee4dad73e 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (13m) 30s ago 13m 62.2M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (5m) 30s ago 12m 69.7M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:19:42.911 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (10m) 100s ago 10m 74.4M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:19:42.912 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (10m) 30s ago 10m 75.9M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:19:42.912 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (10m) 100s ago 10m 72.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:19:42.912 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (10m) 30s ago 10m 75.7M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:19:43.054 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 podman[65965]: 2024-02-13 15:19:42.832901317 +0000 UTC m=+0.336024356 container init 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD) 2024-02-13T15:19:43.054 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 podman[65965]: 2024-02-13 15:19:42.835995716 +0000 UTC m=+0.339118757 container start 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-13T15:19:43.055 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 ceph-osd[65984]: -- 172.21.15.98:0/1289170691 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d70a491e0 con 0x557d70a1b000 2024-02-13T15:19:43.055 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 bash[65965]: 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc 2024-02-13T15:19:43.055 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:42 smithi098 systemd[1]: Started Ceph osd.6 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:19:43.317 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-13T15:19:43.318 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 6 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 11 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:19:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:19:43.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[64469]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[64469]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.320 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[67023]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[67023]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:43 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:43 smithi098 ceph-mon[58041]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:43 smithi098 ceph-mon[58041]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:43 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:43 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:43.656 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:19:43.656 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "11/23 daemons upgraded", 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:19:43.657 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:19:43.755 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:19:43.387+0000 7f801a9487c0 -1 Falling back to public interface 2024-02-13T15:19:44.061 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded 2024-02-13T15:19:44.061 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-13T15:19:44.061 INFO:teuthology.orchestra.run.smithi002.stdout: osd.6 (root=default,host=smithi098) is down 2024-02-13T15:19:44.061 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded 2024-02-13T15:19:44.061 INFO:teuthology.orchestra.run.smithi002.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1d is active+undersized+degraded, acting [2,0] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-02-13T15:19:44.062 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-02-13T15:19:44.063 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-02-13T15:19:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:44 smithi098 ceph-mon[58041]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:44 smithi098 ceph-mon[58041]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:44 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:44.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:44 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3110024183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:44.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:44 smithi098 ceph-mon[58041]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:44.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[67023]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:44.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[67023]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:44.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3110024183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[67023]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[64469]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[64469]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3110024183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:44.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:44 smithi002 ceph-mon[64469]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:19:45.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:45 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/385395360' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:45.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:45 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/385395360' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:45.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:45 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/385395360' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:19:46.305 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:46 smithi098 ceph-mon[58041]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:46.305 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:19:46.095+0000 7f801a9487c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-13T15:19:46.305 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:19:46.124+0000 7f801a9487c0 -1 osd.6 134 log_to_monitors true 2024-02-13T15:19:46.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:46 smithi002 ceph-mon[64469]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:46.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:46 smithi002 ceph-mon[67023]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:47.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:47 smithi098 ceph-mon[58041]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:47 smithi098 ceph-mon[58041]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:47 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:47.144 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[64469]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[64469]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[67023]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[67023]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:47.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:19:47.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:19:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:19:47.556+0000 7f8007d98700 -1 osd.6 134 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:19:48.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: pgmap v185: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: failed to encode map e137 with expected crc 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: osdmap e137: 8 total, 7 up, 8 in 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:48 smithi098 ceph-mon[58041]: from='osd.6 ' entity='osd.6' 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: pgmap v185: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: failed to encode map e137 with expected crc 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: osdmap e137: 8 total, 7 up, 8 in 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[64469]: from='osd.6 ' entity='osd.6' 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: pgmap v185: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: failed to encode map e137 with expected crc 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: osdmap e137: 8 total, 7 up, 8 in 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:48 smithi002 ceph-mon[67023]: from='osd.6 ' entity='osd.6' 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: failed to encode map e138 with expected crc 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458] boot 2024-02-13T15:19:49.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: osdmap e138: 8 total, 8 up, 8 in 2024-02-13T15:19:49.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:49 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:19:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:49.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: failed to encode map e138 with expected crc 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458] boot 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: osdmap e138: 8 total, 8 up, 8 in 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: failed to encode map e138 with expected crc 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: osd.6 [v2:172.21.15.98:6816/1581337458,v1:172.21.15.98:6817/1581337458] boot 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: osdmap e138: 8 total, 8 up, 8 in 2024-02-13T15:19:49.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T15:19:50.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:49] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-02-13T15:19:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:50 smithi098 ceph-mon[58041]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:50 smithi098 ceph-mon[58041]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:50 smithi098 ceph-mon[58041]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:50 smithi098 ceph-mon[58041]: osdmap e139: 8 total, 8 up, 8 in 2024-02-13T15:19:50.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:50 smithi098 ceph-mon[58041]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[64469]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:50.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[64469]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[64469]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[64469]: osdmap e139: 8 total, 8 up, 8 in 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[64469]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[67023]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[67023]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[67023]: failed to encode map e139 with expected crc 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[67023]: osdmap e139: 8 total, 8 up, 8 in 2024-02-13T15:19:50.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:50 smithi002 ceph-mon[67023]: failed to encode map e139 with expected crc 2024-02-13T15:19:52.376 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:19:52.118+0000 7fe20d50d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: pgmap v190: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:52.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.833 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:52 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: pgmap v190: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:52.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:52.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: pgmap v190: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-02-13T15:19:52.879 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:19:52.880 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:52 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:53 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:53 smithi098 ceph-mon[58041]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-13T15:19:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:53 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:53 smithi002 ceph-mon[64469]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-13T15:19:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:53 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:19:53.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:53 smithi002 ceph-mon[67023]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-13T15:19:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:54 smithi098 ceph-mon[58041]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:19:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:54 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded) 2024-02-13T15:19:54.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:54 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:19:54.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[64469]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:19:54.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded) 2024-02-13T15:19:54.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:19:54.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[67023]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-13T15:19:54.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded) 2024-02-13T15:19:54.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:54 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:19:56.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:56 smithi098 ceph-mon[58041]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:19:56.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:56 smithi002 ceph-mon[64469]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:19:56.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:56 smithi002 ceph-mon[67023]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-02-13T15:19:58.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:58 smithi098 ceph-mon[58041]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:19:58.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:58 smithi002 ceph-mon[64469]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:19:58.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:58 smithi002 ceph-mon[67023]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:19:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:19:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:59.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:19:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:19:59.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:19:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:00.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:19:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:19:59] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-02-13T15:20:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:00 smithi002 ceph-mon[64469]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:00.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:00 smithi002 ceph-mon[64469]: overall HEALTH_OK 2024-02-13T15:20:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:00 smithi002 ceph-mon[67023]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:00.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:00 smithi002 ceph-mon[67023]: overall HEALTH_OK 2024-02-13T15:20:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:00 smithi098 ceph-mon[58041]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:00 smithi098 ceph-mon[58041]: overall HEALTH_OK 2024-02-13T15:20:03.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:02 smithi098 ceph-mon[58041]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-13T15:20:03.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:03.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:03.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[64469]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-13T15:20:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:03.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[67023]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 812 B/s rd, 0 op/s 2024-02-13T15:20:03.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:03.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:03.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:04 smithi098 ceph-mon[58041]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:05.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:04 smithi002 ceph-mon[64469]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:05.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:04 smithi002 ceph-mon[67023]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:07.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:06 smithi098 ceph-mon[58041]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:07.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:06 smithi002 ceph-mon[64469]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:07.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:06 smithi002 ceph-mon[67023]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:08.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:07 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:08.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:07 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:08.092 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:07 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:08 smithi098 ceph-mon[58041]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:08 smithi098 ceph-mon[58041]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:08 smithi098 ceph-mon[58041]: Upgrade: osd.7 is safe to restart 2024-02-13T15:20:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[64469]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[64469]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[64469]: Upgrade: osd.7 is safe to restart 2024-02-13T15:20:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[67023]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[67023]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-13T15:20:09.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:08 smithi002 ceph-mon[67023]: Upgrade: osd.7 is safe to restart 2024-02-13T15:20:10.068 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:09 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:10.068 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:09 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:10.068 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:09 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:20:10.068 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:09 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:20:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:10.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:09] "GET /metrics HTTP/1.1" 200 38517 "" "Prometheus/2.33.4" 2024-02-13T15:20:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T15:20:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:09 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:10 smithi098 ceph-mon[58041]: Upgrade: Updating osd.7 2024-02-13T15:20:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:10 smithi098 ceph-mon[58041]: Deploying daemon osd.7 on smithi098 2024-02-13T15:20:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:10 smithi098 ceph-mon[58041]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[64469]: Upgrade: Updating osd.7 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[64469]: Deploying daemon osd.7 on smithi098 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[64469]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[67023]: Upgrade: Updating osd.7 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[67023]: Deploying daemon osd.7 on smithi098 2024-02-13T15:20:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:10 smithi002 ceph-mon[67023]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:12.332 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:12 smithi098 systemd[1]: Stopping Ceph osd.7 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:20:12.747 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[37833]: 2024-02-13T15:20:12.472+0000 7fbc4283a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:20:12.747 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[37833]: 2024-02-13T15:20:12.472+0000 7fbc4283a700 -1 osd.7 139 *** Got signal Terminated *** 2024-02-13T15:20:12.747 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[37833]: 2024-02-13T15:20:12.472+0000 7fbc4283a700 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:20:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:12 smithi098 ceph-mon[58041]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:13.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:12 smithi098 ceph-mon[58041]: osd.7 marked itself down and dead 2024-02-13T15:20:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:12 smithi002 ceph-mon[64469]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:12 smithi002 ceph-mon[64469]: osd.7 marked itself down and dead 2024-02-13T15:20:13.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:12 smithi002 ceph-mon[67023]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T15:20:13.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:12 smithi002 ceph-mon[67023]: osd.7 marked itself down and dead 2024-02-13T15:20:13.391 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:13 smithi098 podman[67878]: 2024-02-13 15:20:13.10794543 +0000 UTC m=+0.882229999 container died d45ed44b2e8cd86a375a0336a434e07fb840b6ca52f01b389b8694f5fb39e1e8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True) 2024-02-13T15:20:13.751 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:13 smithi098 podman[67878]: 2024-02-13 15:20:13.565067023 +0000 UTC m=+1.339351637 container cleanup d45ed44b2e8cd86a375a0336a434e07fb840b6ca52f01b389b8694f5fb39e1e8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, release=754, version=8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-13T15:20:13.751 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:13 smithi098 bash[67878]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7 2024-02-13T15:20:14.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:13 smithi098 ceph-mon[58041]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:20:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:13 smithi098 ceph-mon[58041]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-13T15:20:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:13 smithi098 ceph-mon[58041]: osdmap e140: 8 total, 7 up, 8 in 2024-02-13T15:20:14.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:13 smithi098 ceph-mon[58041]: osdmap e141: 8 total, 7 up, 8 in 2024-02-13T15:20:14.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:13 smithi098 podman[67896]: 2024-02-13 15:20:13.749897825 +0000 UTC m=+0.631134187 container remove d45ed44b2e8cd86a375a0336a434e07fb840b6ca52f01b389b8694f5fb39e1e8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, RELEASE=HEAD, name=centos-stream, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, ceph=True, distribution-scope=public, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[64469]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[64469]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[64469]: osdmap e140: 8 total, 7 up, 8 in 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[64469]: osdmap e141: 8 total, 7 up, 8 in 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[67023]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[67023]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[67023]: osdmap e140: 8 total, 7 up, 8 in 2024-02-13T15:20:14.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:13 smithi002 ceph-mon[67023]: osdmap e141: 8 total, 7 up, 8 in 2024-02-13T15:20:14.429 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.080838603 +0000 UTC m=+0.017897313 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:20:14.429 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.1840876 +0000 UTC m=+0.121146296 container create f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-13T15:20:14.429 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.428719786 +0000 UTC m=+0.365778485 container init f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:20:14.431 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:20:14.727 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.432256314 +0000 UTC m=+0.369315016 container start f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-13T15:20:14.727 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.528763256 +0000 UTC m=+0.465821955 container attach f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS) 2024-02-13T15:20:15.024 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:14 smithi098 ceph-mon[58041]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:15.025 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:14 smithi098 podman[67991]: 2024-02-13 15:20:14.7264592 +0000 UTC m=+0.663517898 container died f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (7m) 2m ago 12m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (6m) 27s ago 11m 55.4M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (6m) 2m ago 10m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (5m) 27s ago 16m 450M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (6m) 2m ago 18m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (5m) 2m ago 18m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (4m) 27s ago 16m 41.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (5m) 2m ago 16m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:20:15.101 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (7m) 2m ago 12m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (6m) 27s ago 12m 19.4M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (4m) 2m ago 16m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (3m) 2m ago 15m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (2m) 2m ago 15m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (2m) 2m ago 15m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (102s) 27s ago 14m 48.7M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (67s) 27s ago 14m 51.8M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed edf5d8026920 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (32s) 27s ago 13m 12.5M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 9608c796f9a7 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (13m) 27s ago 13m 63.8M 2992M 17.2.0 e1d6a67b021e d45ed44b2e8c 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (5m) 27s ago 12m 69.8M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (11m) 2m ago 11m 74.4M - 17.2.0 e1d6a67b021e d8de0a4b3e93 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (11m) 27s ago 11m 76.1M - 17.2.0 e1d6a67b021e b7d0f1eadc1a 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (11m) 2m ago 11m 72.8M - 17.2.0 e1d6a67b021e 2b94d2c05e73 2024-02-13T15:20:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (11m) 27s ago 11m 76.0M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:20:15.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:14 smithi002 ceph-mon[64469]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:15.103 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:14 smithi002 ceph-mon[67023]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 7 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 12 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:20:15.506 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:20:15.760 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 podman[68017]: 2024-02-13 15:20:15.441888089 +0000 UTC m=+0.710300672 container remove f0831ce84d932c5508840b0398af6931c0d99989bd5378fd9c2cd796365bf949 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T15:20:15.760 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-13T15:20:15.760 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 systemd[1]: Stopped Ceph osd.7 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:20:15.760 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.7.service: Consumed 10.242s CPU time. 2024-02-13T15:20:15.761 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 systemd[1]: Starting Ceph osd.7 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:20:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "12/23 daemons upgraded", 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:20:15.846 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:20:16.022 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:15 smithi098 ceph-mon[58041]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.022 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:15 smithi098 ceph-mon[58041]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.022 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:15 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1724801089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:16.022 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 podman[68119]: 2024-02-13 15:20:15.777703107 +0000 UTC m=+0.020024028 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:20:16.022 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:15 smithi098 podman[68119]: 2024-02-13 15:20:15.876641881 +0000 UTC m=+0.118962802 container create f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[64469]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[64469]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1724801089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[67023]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[67023]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:15 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1724801089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:16.248 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-02-13T15:20:16.248 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-13T15:20:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: osd.7 (root=default,host=smithi098) is down 2024-02-13T15:20:16.248 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-13T15:20:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-13T15:20:16.549 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 podman[68119]: 2024-02-13 15:20:16.104625238 +0000 UTC m=+0.346946159 container init f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:20:16.549 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 podman[68119]: 2024-02-13 15:20:16.108260075 +0000 UTC m=+0.350580997 container start f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:20:16.549 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 podman[68119]: 2024-02-13 15:20:16.179379586 +0000 UTC m=+0.421700517 container attach f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-13T15:20:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-mon[58041]: pgmap v204: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:20:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-mon[58041]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:16.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/532266303' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-13T15:20:16.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 bash[68119]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate[68135]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-13T15:20:16.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:16 smithi098 podman[68119]: 2024-02-13 15:20:16.572808752 +0000 UTC m=+0.815129676 container died f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-13T15:20:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[64469]: pgmap v204: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:20:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[64469]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/532266303' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:17.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[67023]: pgmap v204: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T15:20:17.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[67023]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:17.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:16 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/532266303' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:17.520 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 podman[68253]: 2024-02-13 15:20:17.150067054 +0000 UTC m=+0.571883260 container remove f253648f6a888c7878fa8ce081e0e01271581ce28f03fe1c25f354be61ee1e09 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-activate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-13T15:20:17.520 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 podman[68320]: 2024-02-13 15:20:17.266587082 +0000 UTC m=+0.018340176 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:20:17.520 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 podman[68320]: 2024-02-13 15:20:17.393179123 +0000 UTC m=+0.144932260 container create 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-13T15:20:17.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:17 smithi098 ceph-mon[58041]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:17.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:17.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:17.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 podman[68320]: 2024-02-13 15:20:17.62900136 +0000 UTC m=+0.380754460 container init 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:20:17.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 podman[68320]: 2024-02-13 15:20:17.632055085 +0000 UTC m=+0.383808179 container start 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2) 2024-02-13T15:20:17.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 ceph-osd[68340]: -- 172.21.15.98:0/597024145 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dde9f3d1e0 con 0x55dde9f0f000 2024-02-13T15:20:17.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 bash[68320]: 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 2024-02-13T15:20:17.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:17 smithi098 systemd[1]: Started Ceph osd.7 for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:20:18.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[64469]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:18.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:18.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:18.093 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[67023]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:18.094 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:18.094 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:18.568 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:18 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:20:18.177+0000 7f177f1357c0 -1 Falling back to public interface 2024-02-13T15:20:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:18 smithi098 ceph-mon[58041]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 11 stale+active+clean, 29 active+undersized, 102 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 48/627 objects degraded (7.656%) 2024-02-13T15:20:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:18 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:18 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:18.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:18 smithi098 ceph-mon[58041]: Health check failed: Degraded data redundancy: 48/627 objects degraded (7.656%), 19 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[64469]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 11 stale+active+clean, 29 active+undersized, 102 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 48/627 objects degraded (7.656%) 2024-02-13T15:20:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:19.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[64469]: Health check failed: Degraded data redundancy: 48/627 objects degraded (7.656%), 19 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[67023]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 11 stale+active+clean, 29 active+undersized, 102 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 48/627 objects degraded (7.656%) 2024-02-13T15:20:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:19.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:18 smithi002 ceph-mon[67023]: Health check failed: Degraded data redundancy: 48/627 objects degraded (7.656%), 19 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:19.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:19 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:19 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:20.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:19] "GET /metrics HTTP/1.1" 200 38598 "" "Prometheus/2.33.4" 2024-02-13T15:20:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:19 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:21.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:20 smithi098 ceph-mon[58041]: pgmap v206: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:21.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:20 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:20:20.910+0000 7f177f1357c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-13T15:20:21.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:20 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:20:20.945+0000 7f177f1357c0 -1 osd.7 139 log_to_monitors true 2024-02-13T15:20:21.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:20 smithi002 ceph-mon[64469]: pgmap v206: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:21.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:20 smithi002 ceph-mon[67023]: pgmap v206: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:21.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:21 smithi098 ceph-mon[58041]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:21.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:21 smithi098 ceph-mon[58041]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:22.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:21 smithi002 ceph-mon[64469]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:22.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:21 smithi002 ceph-mon[64469]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:22.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:21 smithi002 ceph-mon[67023]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:22.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:21 smithi002 ceph-mon[67023]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T15:20:22.581 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:20:22.258+0000 7f176c585700 -1 osd.7 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T15:20:23.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: pgmap v207: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: osdmap e142: 8 total, 7 up, 8 in 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:22 smithi098 ceph-mon[58041]: from='osd.7 ' entity='osd.7' 2024-02-13T15:20:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: pgmap v207: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:20:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: osdmap e142: 8 total, 7 up, 8 in 2024-02-13T15:20:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[64469]: from='osd.7 ' entity='osd.7' 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: pgmap v207: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: osdmap e142: 8 total, 7 up, 8 in 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:23.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:22 smithi002 ceph-mon[67023]: from='osd.7 ' entity='osd.7' 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961] boot 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: osdmap e143: 8 total, 8 up, 8 in 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:24.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:23 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:20:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961] boot 2024-02-13T15:20:24.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: osdmap e143: 8 total, 8 up, 8 in 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: osd.7 [v2:172.21.15.98:6824/1996576961,v1:172.21.15.98:6825/1996576961] boot 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: osdmap e143: 8 total, 8 up, 8 in 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:24.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:23 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:24 smithi098 ceph-mon[58041]: pgmap v209: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:24 smithi098 ceph-mon[58041]: osdmap e144: 8 total, 8 up, 8 in 2024-02-13T15:20:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:24 smithi002 ceph-mon[64469]: pgmap v209: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:24 smithi002 ceph-mon[64469]: osdmap e144: 8 total, 8 up, 8 in 2024-02-13T15:20:25.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:24 smithi002 ceph-mon[67023]: pgmap v209: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-02-13T15:20:25.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:24 smithi002 ceph-mon[67023]: osdmap e144: 8 total, 8 up, 8 in 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: pgmap v212: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/627 objects degraded (13.078%) 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:27.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:26 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: pgmap v212: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/627 objects degraded (13.078%) 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: pgmap v212: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/627 objects degraded (13.078%) 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.129 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:27.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:26 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all osd 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:28.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:27 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:28.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all osd 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:28.104 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all osd 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-13T15:20:28.105 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-13T15:20:28.106 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:28.106 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:27 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-13T15:20:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: Upgrade: Setting require_osd_release to 19 squid 2024-02-13T15:20:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: pgmap v213: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/627 objects degraded (1.914%) 2024-02-13T15:20:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-13T15:20:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-13T15:20:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: osdmap e145: 8 total, 8 up, 8 in 2024-02-13T15:20:29.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:29.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:28 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: Upgrade: Setting require_osd_release to 19 squid 2024-02-13T15:20:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: pgmap v213: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/627 objects degraded (1.914%) 2024-02-13T15:20:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-13T15:20:29.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: osdmap e145: 8 total, 8 up, 8 in 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: Upgrade: Setting require_osd_release to 19 squid 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: pgmap v213: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/627 objects degraded (1.914%) 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: osdmap e145: 8 total, 8 up, 8 in 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:29.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:28 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:29.811 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all mds 2024-02-13T15:20:29.812 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all mds 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:29 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:29] "GET /metrics HTTP/1.1" 200 38598 "" "Prometheus/2.33.4" 2024-02-13T15:20:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all mds 2024-02-13T15:20:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.jwgbut", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:29 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:30 smithi098 ceph-mon[58041]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:30 smithi098 ceph-mon[58041]: Upgrade: Updating rgw.foo.smithi002.jwgbut (1/4) 2024-02-13T15:20:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:30 smithi098 ceph-mon[58041]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:20:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:30 smithi098 ceph-mon[58041]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded) 2024-02-13T15:20:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:30 smithi098 ceph-mon[58041]: Cluster is now healthy 2024-02-13T15:20:31.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[64469]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[64469]: Upgrade: Updating rgw.foo.smithi002.jwgbut (1/4) 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[64469]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[64469]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded) 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[64469]: Cluster is now healthy 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[67023]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[67023]: Upgrade: Updating rgw.foo.smithi002.jwgbut (1/4) 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[67023]: Deploying daemon rgw.foo.smithi002.jwgbut on smithi002 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[67023]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded) 2024-02-13T15:20:31.084 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:30 smithi002 ceph-mon[67023]: Cluster is now healthy 2024-02-13T15:20:32.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:31 smithi002 ceph-mon[64469]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-13T15:20:32.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:31 smithi002 ceph-mon[67023]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-13T15:20:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:31 smithi098 ceph-mon[58041]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-13T15:20:33.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:33.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:33.306 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:33.306 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:33.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:34.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:33 smithi002 ceph-mon[64469]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:34.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:33 smithi002 ceph-mon[67023]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:34.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:33 smithi098 ceph-mon[58041]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:20:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:34 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:34 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:34 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:35.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:34 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:34 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:35.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:34 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:20:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: Upgrade: Updating rgw.smpl.smithi002.szwgvn (2/4) 2024-02-13T15:20:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:37.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:37.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:36 smithi098 ceph-mon[58041]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: Upgrade: Updating rgw.smpl.smithi002.szwgvn (2/4) 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[64469]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: Upgrade: Updating rgw.smpl.smithi002.szwgvn (2/4) 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.szwgvn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:37.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:36 smithi002 ceph-mon[67023]: Deploying daemon rgw.smpl.smithi002.szwgvn on smithi002 2024-02-13T15:20:38.090 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:38 smithi002 ceph-mon[64469]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 102 B/s wr, 82 op/s 2024-02-13T15:20:38.090 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:38 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:38.090 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:38 smithi002 ceph-mon[67023]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 102 B/s wr, 82 op/s 2024-02-13T15:20:38.091 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:38 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:38 smithi098 ceph-mon[58041]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 102 B/s wr, 82 op/s 2024-02-13T15:20:38.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:38 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:39.295 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:39 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:39.295 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:39 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:39.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:39 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:39.977 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:39] "GET /metrics HTTP/1.1" 200 38606 "" "Prometheus/2.33.4" 2024-02-13T15:20:40.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:40 smithi098 ceph-mon[58041]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 92 B/s wr, 91 op/s 2024-02-13T15:20:40.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:40 smithi002 ceph-mon[64469]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 92 B/s wr, 91 op/s 2024-02-13T15:20:40.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:40 smithi002 ceph-mon[67023]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 92 B/s wr, 91 op/s 2024-02-13T15:20:41.160 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:41 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:41.160 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:41 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:41.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:41 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:41.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:41 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:41.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:41 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:41.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:41 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:42.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:42 smithi098 ceph-mon[58041]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:42.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:42 smithi002 ceph-mon[64469]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:42.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:42 smithi002 ceph-mon[67023]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: Upgrade: Updating rgw.foo.smithi098.eccznt (3/4) 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:43.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:43 smithi098 ceph-mon[58041]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:20:43.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: Upgrade: Updating rgw.foo.smithi098.eccznt (3/4) 2024-02-13T15:20:43.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:43.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[64469]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: Upgrade: Updating rgw.foo.smithi098.eccznt (3/4) 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.eccznt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:43.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:43 smithi002 ceph-mon[67023]: Deploying daemon rgw.foo.smithi098.eccznt on smithi098 2024-02-13T15:20:44.485 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:44 smithi098 ceph-mon[58041]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:44.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:44 smithi002 ceph-mon[64469]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:44.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:44 smithi002 ceph-mon[67023]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:46 smithi098 ceph-mon[58041]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:46 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:46 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[64469]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:46.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.622 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[67023]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:46.622 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.622 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:46 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:46.623 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:20:47.307 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (7m) 2m ago 12m 24.0M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (7m) 25s ago 12m 54.9M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 running (6m) 2m ago 11m 54.3M - 3.5 e1d6a67b021e b6a3eb9cc4a7 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (6m) 25s ago 17m 451M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (7m) 2m ago 18m 505M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (6m) 2m ago 18m 49.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (5m) 25s ago 17m 41.4M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (5m) 2m ago 17m 40.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (7m) 2m ago 13m 18.5M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (7m) 25s ago 13m 19.5M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (4m) 2m ago 16m 45.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 2m ago 16m 51.9M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (3m) 2m ago 15m 54.4M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (2m) 2m ago 15m 14.8M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:20:47.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (2m) 25s ago 15m 50.9M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (99s) 25s ago 14m 52.1M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed edf5d8026920 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (64s) 25s ago 14m 50.4M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 9608c796f9a7 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (30s) 25s ago 14m 12.6M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 00de955572ba 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (6m) 25s ago 13m 69.8M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 starting - - - - 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 starting - - - - 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 starting - - - - 2024-02-13T15:20:47.309 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (11m) 25s ago 11m 76.1M - 17.2.0 e1d6a67b021e 37468866f671 2024-02-13T15:20:47.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:47 smithi098 ceph-mon[58041]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:47.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:47 smithi002 ceph-mon[64469]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:47.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:47.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:47 smithi002 ceph-mon[67023]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:47.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:20:47.711 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 8 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:20:47.712 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 16 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:20:47.713 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:20:48.050 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "mon", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "osd" 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "13/23 daemons upgraded", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading rgw daemons", 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-13T15:20:48.051 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 170 B/s wr, 211 op/s 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2796385770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:48 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 170 B/s wr, 211 op/s 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2796385770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:48.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 170 B/s wr, 211 op/s 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2796385770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi098.kzxzji", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T15:20:48.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:48 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:20:48.462 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-02-13T15:20:49.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:49 smithi098 ceph-mon[58041]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:49.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:49 smithi098 ceph-mon[58041]: Upgrade: Updating rgw.smpl.smithi098.kzxzji (4/4) 2024-02-13T15:20:49.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:49 smithi098 ceph-mon[58041]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:20:49.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:49 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:49.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:49 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3091776897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:49.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[67023]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:49.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[67023]: Upgrade: Updating rgw.smpl.smithi098.kzxzji (4/4) 2024-02-13T15:20:49.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[67023]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:20:49.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:49.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3091776897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:49.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[64469]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:20:49.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[64469]: Upgrade: Updating rgw.smpl.smithi098.kzxzji (4/4) 2024-02-13T15:20:49.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[64469]: Deploying daemon rgw.smpl.smithi098.kzxzji on smithi098 2024-02-13T15:20:49.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:49.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3091776897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:20:49.886 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:49] "GET /metrics HTTP/1.1" 200 38603 "" "Prometheus/2.33.4" 2024-02-13T15:20:50.303 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:50 smithi098 ceph-mon[58041]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 159 op/s 2024-02-13T15:20:50.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:50 smithi002 ceph-mon[64469]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 159 op/s 2024-02-13T15:20:50.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:50 smithi002 ceph-mon[67023]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 159 op/s 2024-02-13T15:20:52.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:52 smithi098 ceph-mon[58041]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 85 B/s wr, 143 op/s 2024-02-13T15:20:52.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:52 smithi002 ceph-mon[64469]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 85 B/s wr, 143 op/s 2024-02-13T15:20:52.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:52 smithi002 ceph-mon[67023]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 85 B/s wr, 143 op/s 2024-02-13T15:20:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:53 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:53.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:53 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:53.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:53 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:53.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:53 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:53.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:53 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:54.350 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:54 smithi002 ceph-mon[64469]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:54.351 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:54 smithi002 ceph-mon[67023]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:54.441 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:54 smithi098 ceph-mon[58041]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-13T15:20:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:56 smithi098 ceph-mon[58041]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:56 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[67023]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:56.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[64469]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:20:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:56.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:56 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:58 smithi098 ceph-mon[58041]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 153 op/s 2024-02-13T15:20:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:58 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[64469]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 153 op/s 2024-02-13T15:20:58.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[67023]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 153 op/s 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:58.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:58 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:20:59.462 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:20:59 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:59.462 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:20:59 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:20:59.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:20:59 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:00.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:20:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:20:59] "GET /metrics HTTP/1.1" 200 38603 "" "Prometheus/2.33.4" 2024-02-13T15:21:00.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:00 smithi098 ceph-mon[58041]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 125 op/s 2024-02-13T15:21:00.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:00 smithi002 ceph-mon[64469]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 125 op/s 2024-02-13T15:21:00.607 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:00 smithi002 ceph-mon[67023]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 125 op/s 2024-02-13T15:21:02.258 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-02-13T15:21:02.258 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: Detected new or changed devices on smithi002 2024-02-13T15:21:02.258 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.258 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: Detected new or changed devices on smithi098 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all rgw 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]': finished 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]': finished 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]': finished 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]': finished 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all rbd-mirror 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.261 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: Upgrade: Setting container_image for all ceph-exporter 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: Detected new or changed devices on smithi002 2024-02-13T15:21:02.262 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: Detected new or changed devices on smithi098 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.263 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all rgw 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]': finished 2024-02-13T15:21:02.264 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]': finished 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]': finished 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]': finished 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.265 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all rbd-mirror 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: Upgrade: Setting container_image for all ceph-exporter 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.266 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:02.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-02-13T15:21:02.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: Detected new or changed devices on smithi002 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: Detected new or changed devices on smithi098 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.333 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all rgw 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.jwgbut"}]': finished 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]: dispatch 2024-02-13T15:21:02.334 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.eccznt"}]': finished 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.szwgvn"}]': finished 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi098.kzxzji"}]': finished 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all rbd-mirror 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:02.335 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: Upgrade: Setting container_image for all ceph-exporter 2024-02-13T15:21:02.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:02.336 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 170 B/s wr, 110 op/s 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: Upgrade: Updating iscsi.foo.smithi002.fozvhf 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:04.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[64469]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:21:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 170 B/s wr, 110 op/s 2024-02-13T15:21:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: Upgrade: Updating iscsi.foo.smithi002.fozvhf 2024-02-13T15:21:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:04.378 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:04.379 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:04 smithi002 ceph-mon[67023]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 170 B/s wr, 110 op/s 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: Upgrade: Updating iscsi.foo.smithi002.fozvhf 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.fozvhf", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:04.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:04 smithi098 ceph-mon[58041]: Deploying daemon iscsi.foo.smithi002.fozvhf on smithi002 2024-02-13T15:21:06.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:06 smithi098 ceph-mon[58041]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:21:06.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:06 smithi002 ceph-mon[64469]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:21:06.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:06 smithi002 ceph-mon[67023]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-13T15:21:08.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:08 smithi098 ceph-mon[58041]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-13T15:21:08.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:08 smithi002 ceph-mon[64469]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-13T15:21:08.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:08 smithi002 ceph-mon[67023]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-13T15:21:09.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:09 smithi098 ceph-mon[58041]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:09.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:09 smithi002 ceph-mon[64469]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:09.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:09 smithi002 ceph-mon[67023]: from='client.25267 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:10.186 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:09] "GET /metrics HTTP/1.1" 200 38672 "" "Prometheus/2.33.4" 2024-02-13T15:21:10.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:10 smithi098 ceph-mon[58041]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:21:10.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:10 smithi002 ceph-mon[64469]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:21:10.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:10 smithi002 ceph-mon[67023]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T15:21:12.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:12 smithi098 ceph-mon[58041]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:12.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:12 smithi002 ceph-mon[64469]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:12.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:12 smithi002 ceph-mon[67023]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:14.581 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:14 smithi098 ceph-mon[58041]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:14.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:14 smithi002 ceph-mon[67023]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:14.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:14 smithi002 ceph-mon[64469]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:16.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:16 smithi098 ceph-mon[58041]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:16.588 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:16 smithi002 ceph-mon[64469]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:16.589 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:16 smithi002 ceph-mon[67023]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:17.218 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:16 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:21:16.876+0000 7fe1ddca5700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:21:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[64469]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:21:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[64469]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[67023]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[67023]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:18.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:17 smithi098 ceph-mon[58041]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:21:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:17 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:17 smithi098 ceph-mon[58041]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:18.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:18.854 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:21:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:18 smithi002 ceph-mon[64469]: Health check failed: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-13T15:21:19.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:18 smithi002 ceph-mon[67023]: Health check failed: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-13T15:21:19.245 INFO:teuthology.orchestra.run.smithi002.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed." 2024-02-13T15:21:19.265 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:18 smithi098 ceph-mon[58041]: Health check failed: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-13T15:21:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[64469]: from='client.44614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[64469]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:20.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[64469]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:20.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:19] "GET /metrics HTTP/1.1" 200 38673 "" "Prometheus/2.33.4" 2024-02-13T15:21:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[67023]: from='client.44614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[67023]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:20.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:19 smithi002 ceph-mon[67023]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:20.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch ps' 2024-02-13T15:21:20.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:19 smithi098 ceph-mon[58041]: from='client.44614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:20.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:19 smithi098 ceph-mon[58041]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:20.279 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:19 smithi098 ceph-mon[58041]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:21.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:21 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:21.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:21 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:21.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:21 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:21.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:21 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:22.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:21 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:22.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:21 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:22.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:22 smithi002 ceph-mon[64469]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:22.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:22 smithi002 ceph-mon[67023]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:23.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:22 smithi098 ceph-mon[58041]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (8m) 0s ago 13m 24.1M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (7m) 2s ago 12m 54.9M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 unknown 0s ago 12m - - 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (7m) 2s ago 17m 451M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (7m) 0s ago 19m 514M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (6m) 0s ago 19m 58.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (6m) 2s ago 17m 43.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:21:23.330 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (6m) 0s ago 18m 48.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (8m) 0s ago 13m 18.4M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (7m) 2s ago 13m 19.2M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (5m) 0s ago 17m 55.5M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 0s ago 16m 64.6M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (4m) 0s ago 16m 62.1M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (3m) 0s ago 16m 66.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (2m) 2s ago 15m 56.2M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (2m) 2s ago 15m 57.3M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed edf5d8026920 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (100s) 2s ago 15m 53.7M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 9608c796f9a7 2024-02-13T15:21:23.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (66s) 2s ago 14m 55.8M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 00de955572ba 2024-02-13T15:21:23.332 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (7m) 2s ago 13m 69.9M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:21:23.332 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (49s) 0s ago 12m 75.7M - 19.0.0-1423-ge1b79d5a 28480620e5ed d3a7a1fa1045 2024-02-13T15:21:23.332 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (37s) 2s ago 12m 74.2M - 19.0.0-1423-ge1b79d5a 28480620e5ed 0896cb3df27d 2024-02-13T15:21:23.332 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (43s) 0s ago 12m 77.7M - 19.0.0-1423-ge1b79d5a 28480620e5ed abf37f635615 2024-02-13T15:21:23.332 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (31s) 2s ago 12m 74.2M - 19.0.0-1423-ge1b79d5a 28480620e5ed c0ccfd7604e1 2024-02-13T15:21:24.005 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph versions' 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 395 B/s rd, 0 op/s 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: Checking dashboard <-> RGW credentials 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='client.44626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.582 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 469 B/s rd, 0 op/s 2024-02-13T15:21:24.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:24.583 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:24.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 395 B/s rd, 0 op/s 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: Checking dashboard <-> RGW credentials 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='client.44626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 469 B/s rd, 0 op/s 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 770 B/s rd, 0 op/s 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.613 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 395 B/s rd, 0 op/s 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: Checking dashboard <-> RGW credentials 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='client.44626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 469 B/s rd, 0 op/s 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:24.614 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:26.154 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 8 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 4 2024-02-13T15:21:26.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:21:26.156 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:21:26.156 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 17 2024-02-13T15:21:26.156 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:21:26.156 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:21:26.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:26 smithi002 ceph-mon[64469]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:26.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:26 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/334115584' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:26.740 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:26 smithi002 ceph-mon[67023]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:26.741 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:26 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/334115584' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:26.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-13T15:21:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:26 smithi098 ceph-mon[58041]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:27.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:26 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/334115584' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:21:28.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:28 smithi002 ceph-mon[64469]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:28.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:28 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2391988646' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:21:28.746 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:28 smithi002 ceph-mon[67023]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:28.746 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:28 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2391988646' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:21:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:28 smithi098 ceph-mon[58041]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:29.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:28 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2391988646' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T15:21:29.130 INFO:teuthology.orchestra.run.smithi002.stdout:wait for servicemap items w/ changing names to refresh 2024-02-13T15:21:29.839 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'sleep 60' 2024-02-13T15:21:30.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:29 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/4046307065' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:30.030 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:29 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:29] "GET /metrics HTTP/1.1" 200 38673 "" "Prometheus/2.33.4" 2024-02-13T15:21:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:29 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/4046307065' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:29 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:29 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/4046307065' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:29 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]: dispatch 2024-02-13T15:21:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:30 smithi098 ceph-mon[58041]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:30 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]': finished 2024-02-13T15:21:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:30 smithi098 ceph-mon[58041]: osdmap e146: 8 total, 8 up, 8 in 2024-02-13T15:21:31.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:30 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]: dispatch 2024-02-13T15:21:31.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[64469]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:31.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]': finished 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[64469]: osdmap e146: 8 total, 8 up, 8 in 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]: dispatch 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[67023]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1315168976"}]': finished 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[67023]: osdmap e146: 8 total, 8 up, 8 in 2024-02-13T15:21:31.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:30 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]: dispatch 2024-02-13T15:21:31.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]': finished 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[64469]: osdmap e147: 8 total, 8 up, 8 in 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/682216155' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]': finished 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[67023]: osdmap e147: 8 total, 8 up, 8 in 2024-02-13T15:21:31.742 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/682216155' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:31.743 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:31 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:32.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:31 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/3471061975' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/835040697"}]': finished 2024-02-13T15:21:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:31 smithi098 ceph-mon[58041]: osdmap e147: 8 total, 8 up, 8 in 2024-02-13T15:21:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:31 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/682216155' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:32.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:31 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]: dispatch 2024-02-13T15:21:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]': finished 2024-02-13T15:21:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: osdmap e148: 8 total, 8 up, 8 in 2024-02-13T15:21:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:33.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/248815850' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:33.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:32 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:33.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]': finished 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: osdmap e148: 8 total, 8 up, 8 in 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/248815850' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2511629966"}]': finished 2024-02-13T15:21:33.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: osdmap e148: 8 total, 8 up, 8 in 2024-02-13T15:21:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:21:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/248815850' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:33.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:32 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]: dispatch 2024-02-13T15:21:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:33 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]': finished 2024-02-13T15:21:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:33 smithi098 ceph-mon[58041]: osdmap e149: 8 total, 8 up, 8 in 2024-02-13T15:21:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:33 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/472026110' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:33 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:33 smithi098 ceph-mon[58041]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-13T15:21:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]': finished 2024-02-13T15:21:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[67023]: osdmap e149: 8 total, 8 up, 8 in 2024-02-13T15:21:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/472026110' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[67023]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-13T15:21:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/587226554"}]': finished 2024-02-13T15:21:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[64469]: osdmap e149: 8 total, 8 up, 8 in 2024-02-13T15:21:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/472026110' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]: dispatch 2024-02-13T15:21:34.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:33 smithi002 ceph-mon[64469]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-02-13T15:21:35.331 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:34 smithi098 ceph-mon[58041]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]': finished 2024-02-13T15:21:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:34 smithi098 ceph-mon[58041]: osdmap e150: 8 total, 8 up, 8 in 2024-02-13T15:21:35.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:34 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]: dispatch 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[67023]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]': finished 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[67023]: osdmap e150: 8 total, 8 up, 8 in 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]: dispatch 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[64469]: from='client.? ' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1545097347"}]': finished 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[64469]: osdmap e150: 8 total, 8 up, 8 in 2024-02-13T15:21:35.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:34 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]: dispatch 2024-02-13T15:21:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:35 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]': finished 2024-02-13T15:21:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:35 smithi098 ceph-mon[58041]: osdmap e151: 8 total, 8 up, 8 in 2024-02-13T15:21:36.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:35 smithi098 ceph-mon[58041]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]': finished 2024-02-13T15:21:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[64469]: osdmap e151: 8 total, 8 up, 8 in 2024-02-13T15:21:36.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[64469]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:36.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2434774258' entity='client.iscsi.foo.smithi002.fozvhf' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2511629966"}]': finished 2024-02-13T15:21:36.376 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[67023]: osdmap e151: 8 total, 8 up, 8 in 2024-02-13T15:21:36.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:35 smithi002 ceph-mon[67023]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-02-13T15:21:39.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:38 smithi098 ceph-mon[58041]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-02-13T15:21:39.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:38 smithi002 ceph-mon[64469]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-02-13T15:21:39.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:38 smithi002 ceph-mon[67023]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-02-13T15:21:39.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:39 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:40.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:39 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:40.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:39] "GET /metrics HTTP/1.1" 200 38752 "" "Prometheus/2.33.4" 2024-02-13T15:21:40.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:39 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:41.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:40 smithi098 ceph-mon[58041]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-13T15:21:41.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:40 smithi002 ceph-mon[64469]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-13T15:21:41.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:40 smithi002 ceph-mon[67023]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-13T15:21:43.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:42 smithi098 ceph-mon[58041]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:43.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:42 smithi002 ceph-mon[64469]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:43.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:42 smithi002 ceph-mon[67023]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:45.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:44 smithi098 ceph-mon[58041]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:21:45.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:44 smithi002 ceph-mon[64469]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:21:45.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:44 smithi002 ceph-mon[67023]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T15:21:47.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:46 smithi098 ceph-mon[58041]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-13T15:21:47.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:46 smithi002 ceph-mon[64469]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-13T15:21:47.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:46 smithi002 ceph-mon[67023]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-13T15:21:48.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:48.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:48.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:21:49.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:48 smithi098 ceph-mon[58041]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:49.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:48 smithi002 ceph-mon[64469]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:49.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:48 smithi002 ceph-mon[67023]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:50.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:49 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:50.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:49 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:50.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:49 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:21:50.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:49] "GET /metrics HTTP/1.1" 200 38751 "" "Prometheus/2.33.4" 2024-02-13T15:21:51.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:50 smithi098 ceph-mon[58041]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:51.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:50 smithi002 ceph-mon[64469]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:51.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:50 smithi002 ceph-mon[67023]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:53.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:52 smithi098 ceph-mon[58041]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:53.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:52 smithi002 ceph-mon[64469]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:53.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:52 smithi002 ceph-mon[67023]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:55.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:54 smithi098 ceph-mon[58041]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:55.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:54 smithi002 ceph-mon[64469]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:55.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:54 smithi002 ceph-mon[67023]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:57.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:56 smithi098 ceph-mon[58041]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:56 smithi002 ceph-mon[64469]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:57.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:56 smithi002 ceph-mon[67023]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:21:59.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:58 smithi098 ceph-mon[58041]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:59.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:58 smithi002 ceph-mon[64469]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:21:59.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:58 smithi002 ceph-mon[67023]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:00.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:21:59 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:00.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:21:59 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:00.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:21:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:21:59] "GET /metrics HTTP/1.1" 200 38751 "" "Prometheus/2.33.4" 2024-02-13T15:22:00.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:21:59 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:01.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:00 smithi098 ceph-mon[58041]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:01.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:00 smithi002 ceph-mon[64469]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:01.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:00 smithi002 ceph-mon[67023]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:02 smithi098 ceph-mon[58041]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:03.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:02 smithi002 ceph-mon[64469]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:03.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:03.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:02 smithi002 ceph-mon[67023]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:03.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:05.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:04 smithi098 ceph-mon[58041]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:05.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:04 smithi002 ceph-mon[64469]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:05.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:04 smithi002 ceph-mon[67023]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:07.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:06 smithi098 ceph-mon[58041]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:07.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:06 smithi002 ceph-mon[64469]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:07.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:06 smithi002 ceph-mon[67023]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:09.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:08 smithi098 ceph-mon[58041]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:09.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:08 smithi002 ceph-mon[64469]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:09.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:08 smithi002 ceph-mon[67023]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:10.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:09 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:09 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:10.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:09] "GET /metrics HTTP/1.1" 200 38749 "" "Prometheus/2.33.4" 2024-02-13T15:22:10.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:09 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:11.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:10 smithi098 ceph-mon[58041]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:11.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:10 smithi002 ceph-mon[64469]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:11.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:10 smithi002 ceph-mon[67023]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:13.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:12 smithi098 ceph-mon[58041]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:13.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:12 smithi002 ceph-mon[64469]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:13.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:12 smithi002 ceph-mon[67023]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:15.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:14 smithi098 ceph-mon[58041]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:15.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:14 smithi002 ceph-mon[64469]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:15.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:14 smithi002 ceph-mon[67023]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:17.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:16 smithi098 ceph-mon[58041]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:17.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:16 smithi002 ceph-mon[64469]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:17.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:16 smithi002 ceph-mon[67023]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:18.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:17 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:18.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:17 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:18.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:17 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:19.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:18 smithi098 ceph-mon[58041]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:19.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:18 smithi002 ceph-mon[67023]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:19.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:18 smithi002 ceph-mon[64469]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:20.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:19 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:20.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:19 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:20.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:19] "GET /metrics HTTP/1.1" 200 38750 "" "Prometheus/2.33.4" 2024-02-13T15:22:20.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:19 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:21.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:20 smithi098 ceph-mon[58041]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:21.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:20 smithi002 ceph-mon[64469]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:21.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:20 smithi002 ceph-mon[67023]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:23.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:22 smithi098 ceph-mon[58041]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:23.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:22 smithi002 ceph-mon[64469]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:23.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:22 smithi002 ceph-mon[67023]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:24 smithi098 ceph-mon[58041]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:22:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:24 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:22:25.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:24 smithi098 ceph-mon[58041]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[64469]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[64469]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[67023]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T15:22:25.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:24 smithi002 ceph-mon[67023]: from='mgr.25303 ' entity='mgr.y' 2024-02-13T15:22:26.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:25 smithi098 ceph-mon[58041]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-02-13T15:22:26.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:25 smithi098 ceph-mon[58041]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:26.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:25 smithi002 ceph-mon[64469]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-02-13T15:22:26.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:25 smithi002 ceph-mon[64469]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:26.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:25 smithi002 ceph-mon[67023]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-02-13T15:22:26.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:25 smithi002 ceph-mon[67023]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:28.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:27 smithi098 ceph-mon[58041]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:28.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:27 smithi002 ceph-mon[64469]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:28.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:27 smithi002 ceph-mon[67023]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:30.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:29 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:30.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:29 smithi098 ceph-mon[58041]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:29 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:30.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:29 smithi002 ceph-mon[64469]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:30.126 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:29 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:29] "GET /metrics HTTP/1.1" 200 38750 "" "Prometheus/2.33.4" 2024-02-13T15:22:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:29 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:30.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:29 smithi002 ceph-mon[67023]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:31.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:31 smithi002 ceph-mon[64469]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:31.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:31 smithi002 ceph-mon[67023]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:32.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:31 smithi098 ceph-mon[58041]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-13T15:22:32.336 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch ps' 2024-02-13T15:22:32.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:32 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:32.944 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:32 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:33.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:32 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:34.078 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:33 smithi002 ceph-mon[64469]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:34.078 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:33 smithi002 ceph-mon[67023]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:34.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:33 smithi098 ceph-mon[58041]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (9m) 71s ago 14m 24.1M - 0.23.0 ba2b418f427c a9e94646913c 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi098 *:3000 running (8m) 73s ago 14m 54.9M - 8.3.5 dad864ee21e9 f31dbe3f643a 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.fozvhf smithi002 unknown 71s ago 13m - - 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi098 *:8443,9283,8765 running (8m) 73s ago 18m 451M - 19.0.0-1423-ge1b79d5a 28480620e5ed 8010b85e5706 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (8m) 71s ago 20m 514M - 19.0.0-1423-ge1b79d5a 28480620e5ed 927869768fbd 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (7m) 71s ago 20m 58.2M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 304db26dc9b1 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi098 running (7m) 73s ago 19m 43.6M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed aaab9e770145 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (7m) 71s ago 19m 48.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed 6c133270cc54 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (9m) 71s ago 15m 18.4M - 1.3.1 1dbe0e931976 9e47fb0e6796 2024-02-13T15:22:34.561 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi098 *:9100 running (9m) 73s ago 14m 19.2M - 1.3.1 1dbe0e931976 1735010684f0 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 71s ago 18m 55.5M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed ae4ea389aea7 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (5m) 71s ago 18m 64.6M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed c823921c4f44 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (5m) 71s ago 17m 62.1M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed d413178468c8 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (4m) 71s ago 17m 66.0M 2737M 19.0.0-1423-ge1b79d5a 28480620e5ed e8dfbfe7cd25 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi098 running (4m) 73s ago 17m 56.2M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 185eb10048c4 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi098 running (3m) 73s ago 16m 57.3M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed edf5d8026920 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi098 running (2m) 73s ago 16m 53.7M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 9608c796f9a7 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi098 running (2m) 73s ago 15m 55.8M 2992M 19.0.0-1423-ge1b79d5a 28480620e5ed 00de955572ba 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi098 *:9095 running (8m) 73s ago 15m 69.9M - 2.33.4 514e6a882f6e 08bdc7bdc6b4 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.jwgbut smithi002 *:8000 running (2m) 71s ago 13m 75.7M - 19.0.0-1423-ge1b79d5a 28480620e5ed d3a7a1fa1045 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi098.eccznt smithi098 *:8000 running (109s) 73s ago 13m 74.2M - 19.0.0-1423-ge1b79d5a 28480620e5ed 0896cb3df27d 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.szwgvn smithi002 *:80 running (114s) 71s ago 13m 77.7M - 19.0.0-1423-ge1b79d5a 28480620e5ed abf37f635615 2024-02-13T15:22:34.562 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi098.kzxzji smithi098 *:80 running (102s) 73s ago 13m 74.2M - 19.0.0-1423-ge1b79d5a 28480620e5ed c0ccfd7604e1 2024-02-13T15:22:35.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph versions' 2024-02-13T15:22:36.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:35 smithi098 ceph-mon[58041]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:22:36.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:35 smithi098 ceph-mon[58041]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:36.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:35 smithi002 ceph-mon[64469]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:22:36.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:35 smithi002 ceph-mon[64469]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:36.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:35 smithi002 ceph-mon[67023]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T15:22:36.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:35 smithi002 ceph-mon[67023]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:37.425 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 3 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 2 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 8 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 4 2024-02-13T15:22:37.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-13T15:22:37.427 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-13T15:22:37.427 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)": 17 2024-02-13T15:22:37.427 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-13T15:22:37.427 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:22:38.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:37 smithi002 ceph-mon[64469]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:38.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:37 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1056947424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:38.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:37 smithi002 ceph-mon[67023]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:38.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:37 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1056947424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:38.054 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch upgrade status' 2024-02-13T15:22:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:37 smithi098 ceph-mon[58041]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:38.082 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:37 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1056947424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:38.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:38 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:38.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:38 smithi002 ceph-mon[64469]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:38 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:38.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:38 smithi002 ceph-mon[67023]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:38 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:39.332 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:38 smithi098 ceph-mon[58041]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:40.112 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:39] "GET /metrics HTTP/1.1" 200 38748 "" "Prometheus/2.33.4" 2024-02-13T15:22:40.112 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:22:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662", 2024-02-13T15:22:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-13T15:22:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-13T15:22:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mon", 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw", 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "osd" 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "17/23 daemons upgraded", 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed.", 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": true 2024-02-13T15:22:40.113 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:22:40.796 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph health detail' 2024-02-13T15:22:41.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:41 smithi002 ceph-mon[64469]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:41.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:41 smithi002 ceph-mon[64469]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:41.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:41 smithi002 ceph-mon[67023]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:41.627 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:41 smithi002 ceph-mon[67023]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:41.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:41 smithi098 ceph-mon[58041]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:41.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:41 smithi098 ceph-mon[58041]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout: daemon iscsi.foo.smithi002.fozvhf on smithi002 is in unknown state 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.fozvhf on host smithi002 failed. 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrade daemon: iscsi.foo.smithi002.fozvhf: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi002.fozvhf ... 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:Creating ceph-iscsi config... 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/iscsi-gateway.cfg 2024-02-13T15:22:42.989 INFO:teuthology.orchestra.run.smithi002.stdout:Write file: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/iscsi.foo.smithi002.fozvhf/tcmu-runner-entrypoint.sh 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2dc0fca16a\x2dca80\x2d11ee\x2d95b9\x2d87774f69a715.slice/ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl start ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout:Traceback (most recent call last): 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: return _run_code(code, main_globals, None, 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: exec(code, run_globals) 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5527, in 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 5515, in main 2024-02-13T15:22:42.990 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3045, in command_deploy_from 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3080, in _common_deploy 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 3100, in _deploy_daemon_container 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1036, in deploy_daemon 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/__main__.py", line 1172, in deploy_daemon_units 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout: File "/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/cephadm.8bb8f96bc63665594298f68279dfcabe3174fd3494423270899f969803e5225d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout:RuntimeError: Failed command: systemctl start ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf: Job for ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service failed because the control process exited with error code. 2024-02-13T15:22:42.991 INFO:teuthology.orchestra.run.smithi002.stdout:See "systemctl status ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" and "journalctl -xeu ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@iscsi.foo.smithi002.fozvhf.service" for details. 2024-02-13T15:22:43.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:43 smithi002 ceph-mon[64469]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:43.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:43 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1169490821' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:22:43.611 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:43 smithi002 ceph-mon[67023]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:43.612 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:43 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1169490821' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:22:43.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-13T15:22:43.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:43 smithi098 ceph-mon[58041]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:43.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:43 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1169490821' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-13T15:22:45.694 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-13T15:22:45.831 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:45 smithi002 ceph-mon[64469]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:45.831 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:45 smithi002 ceph-mon[67023]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:45.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:45 smithi098 ceph-mon[58041]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:46.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-13T15:22:46.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:46 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/2458076723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:46.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:46 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/2458076723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:46.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:46 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/2458076723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:47.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:47 smithi002 ceph-mon[64469]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:47.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:47 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:47.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:47 smithi002 ceph-mon[67023]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:47.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:47 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:47.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:47 smithi098 ceph-mon[58041]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:47.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:47 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:22:48.660 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)" 2024-02-13T15:22:49.480 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -e sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-13T15:22:49.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:49.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[64469]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:49.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[64469]: from='client.? 172.21.15.2:0/1928560485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:49.797 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:49.797 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[67023]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:49.797 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-mon[67023]: from='client.? 172.21.15.2:0/1928560485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:49.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:49 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:49.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:49 smithi098 ceph-mon[58041]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:49.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:49 smithi098 ceph-mon[58041]: from='client.? 172.21.15.2:0/1928560485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-13T15:22:50.091 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:49 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:49] "GET /metrics HTTP/1.1" 200 38748 "" "Prometheus/2.33.4" 2024-02-13T15:22:51.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:51 smithi098 ceph-mon[58041]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:51.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:51 smithi002 ceph-mon[64469]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:51.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:51 smithi002 ceph-mon[67023]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:51.877 INFO:teuthology.orchestra.run.smithi002.stdout:osd 8 91s ago - 2024-02-13T15:22:52.575 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-13T15:22:52.584 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T15:22:52.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-13T15:22:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:53 smithi098 ceph-mon[58041]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:53.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:53 smithi098 ceph-mon[58041]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:53 smithi002 ceph-mon[64469]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:53.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:53 smithi002 ceph-mon[64469]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:53.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:53 smithi002 ceph-mon[67023]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:53.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:53 smithi002 ceph-mon[67023]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:55 smithi098 ceph-mon[58041]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:55.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:55 smithi098 ceph-mon[58041]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:55 smithi002 ceph-mon[64469]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:55.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:55 smithi002 ceph-mon[64469]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:55.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:55 smithi002 ceph-mon[67023]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:55.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:55 smithi002 ceph-mon[67023]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout: "image": "quay.io/ceph/ceph", 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout: "registry": "quay.io", 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout: "bare_image": "ceph/ceph", 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout: "versions": [] 2024-02-13T15:22:56.042 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-13T15:22:56.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-02-13T15:22:57.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:57 smithi098 ceph-mon[58041]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:57.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:57 smithi002 ceph-mon[64469]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:57.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:57 smithi002 ceph-mon[67023]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:22:59.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:59 smithi098 ceph-mon[58041]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:59.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:22:59 smithi098 ceph-mon[58041]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:59 smithi002 ceph-mon[64469]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:59.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:22:59 smithi002 ceph-mon[64469]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:22:59.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:59 smithi002 ceph-mon[67023]: from='client.34764 -' entity='client.iscsi.foo.smithi002.fozvhf' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T15:22:59.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:22:59 smithi002 ceph-mon[67023]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:23:00.145 INFO:teuthology.orchestra.run.smithi002.stdout: "16.2.0", 2024-02-13T15:23:00.146 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:22:59 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: ::ffff:172.21.15.98 - - [13/Feb/2024:15:22:59] "GET /metrics HTTP/1.1" 200 38748 "" "Prometheus/2.33.4" 2024-02-13T15:23:00.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:00 smithi098 ceph-mon[58041]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:00.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:00 smithi002 ceph-mon[64469]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:00.875 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:00 smithi002 ceph-mon[67023]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:00.904 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-13T15:23:01.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:01 smithi098 ceph-mon[58041]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:01.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:01 smithi002 ceph-mon[64469]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:01.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:01 smithi002 ceph-mon[67023]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:02.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:02 smithi002 ceph-mon[64469]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:23:02.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:02 smithi002 ceph-mon[67023]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:23:02.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:02 smithi098 ceph-mon[58041]: from='mgr.25303 172.21.15.2:0/2871257912' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T15:23:03.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:03 smithi098 ceph-mon[58041]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:23:03.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:03 smithi002 ceph-mon[64469]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:23:03.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:03 smithi002 ceph-mon[67023]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T15:23:04.404 INFO:teuthology.orchestra.run.smithi002.stdout: "v16.2.2", 2024-02-13T15:23:04.405 INFO:teuthology.orchestra.run.smithi002.stdout: "v16.2.2-20210505", 2024-02-13T15:23:04.700 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:04 smithi002 ceph-mon[64469]: from='client.44788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:04.701 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:04 smithi002 ceph-mon[67023]: from='client.44788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:04.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:04 smithi098 ceph-mon[58041]: from='client.44788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T15:23:05.113 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-13T15:23:05.122 INFO:tasks.cephadm:Teardown begin 2024-02-13T15:23:05.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:23:05.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:23:05.191 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-13T15:23:05.191 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T15:23:05.211 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T15:23:05.248 INFO:tasks.cephadm:Stopping all daemons... 2024-02-13T15:23:05.248 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-13T15:23:05.248 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a 2024-02-13T15:23:05.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:05 smithi002 systemd[1]: Stopping Ceph mon.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:05.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:05 smithi098 ceph-mon[58041]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:05.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:05 smithi002 ceph-mon[67023]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:05 smithi002 ceph-mon[64469]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T15:23:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[64465]: 2024-02-13T15:23:05.553+0000 7f2d26799700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:05 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a[64465]: 2024-02-13T15:23:05.553+0000 7f2d26799700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-13T15:23:05.876 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:05 smithi002 podman[91139]: 2024-02-13 15:23:05.789071419 +0000 UTC m=+0.349019210 container died 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-13T15:23:06.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:06 smithi002 podman[91139]: 2024-02-13 15:23:06.246392565 +0000 UTC m=+0.806340363 container cleanup 304db26dc9b1e5cbf19c5478c68b1aa9540550c7b4bf2895b6ddfb14c209fd59 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-13T15:23:06.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Feb 13 15:23:06 smithi002 bash[91139]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-a 2024-02-13T15:23:06.674 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.a.service' 2024-02-13T15:23:06.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:06.726 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-13T15:23:06.726 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-13T15:23:06.727 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.c 2024-02-13T15:23:06.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:06 smithi002 systemd[1]: Stopping Ceph mon.c for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:07.344 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c[67019]: 2024-02-13T15:23:07.000+0000 7fcf0e415700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c[67019]: 2024-02-13T15:23:07.000+0000 7fcf0e415700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:07 smithi002 podman[91262]: 2024-02-13 15:23:07.04189124 +0000 UTC m=+0.133933032 container died 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE Bus STOPPING 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE Bus STOPPED 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE Bus STARTING 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: CherryPy Checker: 2024-02-13T15:23:07.345 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: The Application mounted at '' has an empty config. 2024-02-13T15:23:07.346 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:23:07.346 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE Serving on http://:::9283 2024-02-13T15:23:07.346 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:07 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: [13/Feb/2024:15:23:07] ENGINE Bus STARTED 2024-02-13T15:23:07.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:07 smithi002 podman[91262]: 2024-02-13 15:23:07.559629259 +0000 UTC m=+0.651671051 container cleanup 6c133270cc54f808a5abf6c244f754f0d0cb26bb36b56d4d66bb2cdebfbc1659 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:07.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Feb 13 15:23:07 smithi002 bash[91262]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-c 2024-02-13T15:23:07.873 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.c.service' 2024-02-13T15:23:07.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:07.909 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-13T15:23:07.909 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-13T15:23:07.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.b 2024-02-13T15:23:08.214 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:07 smithi098 systemd[1]: Stopping Ceph mon.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:08.531 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b[58037]: 2024-02-13T15:23:08.212+0000 7f0f2714f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:08.531 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b[58037]: 2024-02-13T15:23:08.212+0000 7f0f2714f700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-13T15:23:08.531 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:08 smithi098 podman[76552]: 2024-02-13 15:23:08.253732516 +0000 UTC m=+0.146797128 container died aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_CLEAN=True) 2024-02-13T15:23:08.831 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:08 smithi098 podman[76552]: 2024-02-13 15:23:08.645078646 +0000 UTC m=+0.538143250 container cleanup aaab9e7701459b3c7b318af8ba05bd3ada2f10c17a7240218dcdf85e9678840e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:08.832 INFO:journalctl@ceph.mon.b.smithi098.stdout:Feb 13 15:23:08 smithi098 bash[76552]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mon-b 2024-02-13T15:23:09.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mon.b.service' 2024-02-13T15:23:09.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:09.109 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-13T15:23:09.109 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-13T15:23:09.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y 2024-02-13T15:23:09.423 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 systemd[1]: Stopping Ceph mgr.y for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:09.750 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:23:09.422+0000 7fe24bd2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:09.750 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y[60177]: 2024-02-13T15:23:09.422+0000 7fe24bd2f700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-13T15:23:09.750 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 podman[91393]: 2024-02-13 15:23:09.453304137 +0000 UTC m=+0.155129236 container died 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-13T15:23:10.100 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 podman[91393]: 2024-02-13 15:23:09.831673295 +0000 UTC m=+0.533498393 container cleanup 927869768fbdd686f77a082f06d3069b0e5697305f806dd63e11b56ce9ec6bc9 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-13T15:23:10.100 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Feb 13 15:23:09 smithi002 bash[91393]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-y 2024-02-13T15:23:10.249 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-13T15:23:10.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:10.285 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-13T15:23:10.285 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-13T15:23:10.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x 2024-02-13T15:23:10.581 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:23:10 smithi098 systemd[1]: Stopping Ceph mgr.x for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:10.884 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:23:10 smithi098 podman[76674]: 2024-02-13 15:23:10.614429268 +0000 UTC m=+0.136293961 container died 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-13T15:23:11.227 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:23:10 smithi098 podman[76674]: 2024-02-13 15:23:10.958966609 +0000 UTC m=+0.480831319 container cleanup 8010b85e5706c0d556ed54c51f27a69566db791c577faf5ec0851e348507d750 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-13T15:23:11.227 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Feb 13 15:23:10 smithi098 bash[76674]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-mgr-x 2024-02-13T15:23:11.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-13T15:23:11.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:11.373 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-13T15:23:11.373 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-13T15:23:11.373 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.0 2024-02-13T15:23:11.686 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:11 smithi002 systemd[1]: Stopping Ceph osd.0 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:12.126 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:23:11.685+0000 7f6258fbc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:12.126 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:23:11.685+0000 7f6258fbc700 -1 osd.0 151 *** Got signal Terminated *** 2024-02-13T15:23:12.126 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:11 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0[70391]: 2024-02-13T15:23:11.685+0000 7f6258fbc700 -1 osd.0 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:14.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:14.237Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:14.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:14.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:17.331 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:23:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:23:16.888Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-02-13T15:23:17.332 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:23:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:23:16.889Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-02-13T15:23:17.332 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:23:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:23:16.889Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-02-13T15:23:17.332 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:23:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:23:16.891Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-02-13T15:23:17.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:17 smithi002 podman[91516]: 2024-02-13 15:23:17.232167672 +0000 UTC m=+5.670113989 container died ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-13T15:23:17.988 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:17 smithi002 podman[91516]: 2024-02-13 15:23:17.761185834 +0000 UTC m=+6.199132156 container cleanup ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-02-13T15:23:17.988 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:17 smithi002 bash[91516]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0 2024-02-13T15:23:18.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:17 smithi002 podman[91534]: 2024-02-13 15:23:17.987489874 +0000 UTC m=+0.746702272 container remove ae4ea389aea7ac34089d3c3b0b47addbe275a4b31ca53885e60d832a725753de (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:23:18.750 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.313029859 +0000 UTC m=+0.018166970 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:18.750 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.413430442 +0000 UTC m=+0.118567580 container create 071683d5668e5c11646de723028f204b55be1449762ef6ba8379c3c73ea141bb (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-13T15:23:18.750 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.649080145 +0000 UTC m=+0.354217250 container init 071683d5668e5c11646de723028f204b55be1449762ef6ba8379c3c73ea141bb (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:18.750 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.652430001 +0000 UTC m=+0.357567101 container start 071683d5668e5c11646de723028f204b55be1449762ef6ba8379c3c73ea141bb (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-13T15:23:18.750 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.749121755 +0000 UTC m=+0.454258882 container attach 071683d5668e5c11646de723028f204b55be1449762ef6ba8379c3c73ea141bb (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-13T15:23:19.126 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 13 15:23:18 smithi002 podman[91627]: 2024-02-13 15:23:18.947657418 +0000 UTC m=+0.652794502 container died 071683d5668e5c11646de723028f204b55be1449762ef6ba8379c3c73ea141bb (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux ) 2024-02-13T15:23:19.833 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.0.service' 2024-02-13T15:23:19.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:19.904 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-13T15:23:19.904 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-13T15:23:19.904 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.1 2024-02-13T15:23:20.266 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:19 smithi002 systemd[1]: Stopping Ceph osd.1 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:20.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:23:20.264+0000 7f7a41f6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:20.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:23:20.264+0000 7f7a41f6f700 -1 osd.1 151 *** Got signal Terminated *** 2024-02-13T15:23:20.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:20 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1[72869]: 2024-02-13T15:23:20.264+0000 7f7a41f6f700 -1 osd.1 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:24.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:23:24.237Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:24.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:24.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:24.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:24 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:24.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:25.876 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:25 smithi002 podman[91743]: 2024-02-13 15:23:25.6106932 +0000 UTC m=+5.444634586 container died c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:26.258 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91743]: 2024-02-13 15:23:26.014819968 +0000 UTC m=+5.848761350 container cleanup c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:26.258 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 bash[91743]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1 2024-02-13T15:23:26.558 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91763]: 2024-02-13 15:23:26.257658212 +0000 UTC m=+0.646315907 container remove c823921c4f44aff622dfdaf3748f863643002512df83d11cda6fe878e4b4c5d7 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-13T15:23:26.811 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91857]: 2024-02-13 15:23:26.461039311 +0000 UTC m=+0.017663852 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:26.811 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91857]: 2024-02-13 15:23:26.583615127 +0000 UTC m=+0.140239660 container create 8154a4191f96cdb1febe655b0aae7637ee4e87ab99cc14de45ad6912f89a5f5f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:26.811 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91857]: 2024-02-13 15:23:26.811249692 +0000 UTC m=+0.367874225 container init 8154a4191f96cdb1febe655b0aae7637ee4e87ab99cc14de45ad6912f89a5f5f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:23:27.113 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91857]: 2024-02-13 15:23:26.814325258 +0000 UTC m=+0.370949790 container start 8154a4191f96cdb1febe655b0aae7637ee4e87ab99cc14de45ad6912f89a5f5f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-13T15:23:27.113 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:26 smithi002 podman[91857]: 2024-02-13 15:23:26.886300231 +0000 UTC m=+0.442924776 container attach 8154a4191f96cdb1febe655b0aae7637ee4e87ab99cc14de45ad6912f89a5f5f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-13T15:23:27.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 13 15:23:27 smithi002 podman[91857]: 2024-02-13 15:23:27.113221096 +0000 UTC m=+0.669845641 container died 8154a4191f96cdb1febe655b0aae7637ee4e87ab99cc14de45ad6912f89a5f5f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:23:27.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.1.service' 2024-02-13T15:23:27.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:27.923 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-13T15:23:27.923 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-13T15:23:27.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.2 2024-02-13T15:23:28.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:27 smithi002 systemd[1]: Stopping Ceph osd.2 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:28.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:23:28.225+0000 7f3246a52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:28.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:23:28.225+0000 7f3246a52700 -1 osd.2 151 *** Got signal Terminated *** 2024-02-13T15:23:28.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:28 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2[75335]: 2024-02-13T15:23:28.225+0000 7f3246a52700 -1 osd.2 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:34.027 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:33 smithi002 podman[91960]: 2024-02-13 15:23:33.743724334 +0000 UTC m=+5.633662747 container died d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0) 2024-02-13T15:23:34.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:34 smithi002 podman[91960]: 2024-02-13 15:23:34.117841836 +0000 UTC m=+6.007780256 container cleanup d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:34.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:34 smithi002 bash[91960]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2 2024-02-13T15:23:34.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:34 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:23:34.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:34.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:34 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:34.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:34.376 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:34 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:34.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:34.736 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:34 smithi002 podman[91979]: 2024-02-13 15:23:34.402672647 +0000 UTC m=+0.649433777 container remove d413178468c83815c8bdb3e99611be998084440d23062a649afea009822838d5 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:35.006 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:34 smithi002 podman[92069]: 2024-02-13 15:23:34.639186293 +0000 UTC m=+0.018472910 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:35.006 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:34 smithi002 podman[92069]: 2024-02-13 15:23:34.761554993 +0000 UTC m=+0.140841602 container create da8568647f16bb5ef915efd8d3fba8ad38d486a9a0ecd431a8f8c6da03d2a795 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:23:35.006 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:35 smithi002 podman[92069]: 2024-02-13 15:23:35.00563901 +0000 UTC m=+0.384925612 container init da8568647f16bb5ef915efd8d3fba8ad38d486a9a0ecd431a8f8c6da03d2a795 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-13T15:23:35.311 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:35 smithi002 podman[92069]: 2024-02-13 15:23:35.008759197 +0000 UTC m=+0.388045798 container start da8568647f16bb5ef915efd8d3fba8ad38d486a9a0ecd431a8f8c6da03d2a795 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-13T15:23:35.311 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:35 smithi002 podman[92069]: 2024-02-13 15:23:35.088972066 +0000 UTC m=+0.468258677 container attach da8568647f16bb5ef915efd8d3fba8ad38d486a9a0ecd431a8f8c6da03d2a795 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:35.617 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 13 15:23:35 smithi002 podman[92069]: 2024-02-13 15:23:35.311180979 +0000 UTC m=+0.690467590 container died da8568647f16bb5ef915efd8d3fba8ad38d486a9a0ecd431a8f8c6da03d2a795 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-13T15:23:35.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.2.service' 2024-02-13T15:23:35.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:35.952 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-13T15:23:35.952 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-13T15:23:35.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.3 2024-02-13T15:23:36.280 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:36 smithi002 systemd[1]: Stopping Ceph osd.3 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:36.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:36 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:36.277+0000 7f653fdf9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:36.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:36 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:36.277+0000 7f653fdf9700 -1 osd.3 151 *** Got signal Terminated *** 2024-02-13T15:23:36.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:36 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:36.277+0000 7f653fdf9700 -1 osd.3 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:37.773 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:37 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:37.355+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:38.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:37 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:37.771+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:38.581 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:38.381+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:39.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:38 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:38.745+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:39.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:39 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:39.211+0000 7f65383e5700 -1 osd.3 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:14.742676+0000 front 2024-02-13T15:23:14.742855+0000 (oldest deadline 2024-02-13T15:23:38.842784+0000) 2024-02-13T15:23:39.785 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:39.515+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:39.785 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:39.345+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:40.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:39 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:39.784+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:40.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:40 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:40.219+0000 7f65383e5700 -1 osd.3 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:14.742676+0000 front 2024-02-13T15:23:14.742855+0000 (oldest deadline 2024-02-13T15:23:38.842784+0000) 2024-02-13T15:23:40.792 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:40 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:40.499+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:40.792 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:40 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:40.334+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:41.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:40 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:40.790+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:41.626 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:41 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3[77822]: 2024-02-13T15:23:41.255+0000 7f65383e5700 -1 osd.3 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:14.742676+0000 front 2024-02-13T15:23:14.742855+0000 (oldest deadline 2024-02-13T15:23:38.842784+0000) 2024-02-13T15:23:41.815 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:41.471+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:41.815 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:41.364+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:42.022 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:41 smithi002 podman[92173]: 2024-02-13 15:23:41.757584803 +0000 UTC m=+5.579997332 container died e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-13T15:23:42.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:41 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:41.813+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:42.376 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:42 smithi002 podman[92173]: 2024-02-13 15:23:42.119916887 +0000 UTC m=+5.942329415 container cleanup e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-13T15:23:42.376 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:42 smithi002 bash[92173]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3 2024-02-13T15:23:42.738 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:42 smithi002 podman[92191]: 2024-02-13 15:23:42.412757623 +0000 UTC m=+0.655173146 container remove e8dfbfe7cd25226db45afb6f829168cb242dadf5bb24b650ca866a20301e5575 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:23:42.831 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:42.514+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:42.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:42.409+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:43.016 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:42 smithi002 podman[92281]: 2024-02-13 15:23:42.640869765 +0000 UTC m=+0.017768086 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:43.017 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:42 smithi002 podman[92281]: 2024-02-13 15:23:42.738473264 +0000 UTC m=+0.115371588 container create 01e1196601b6be4f70e8b27143b698a1d68f2a04b9bd799d41842f1eec76411d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-13T15:23:43.017 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:43 smithi002 podman[92281]: 2024-02-13 15:23:43.015937035 +0000 UTC m=+0.392835351 container init 01e1196601b6be4f70e8b27143b698a1d68f2a04b9bd799d41842f1eec76411d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-13T15:23:43.323 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:43 smithi002 podman[92281]: 2024-02-13 15:23:43.019417744 +0000 UTC m=+0.396316066 container start 01e1196601b6be4f70e8b27143b698a1d68f2a04b9bd799d41842f1eec76411d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:43.323 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:43 smithi002 podman[92281]: 2024-02-13 15:23:43.099330134 +0000 UTC m=+0.476228464 container attach 01e1196601b6be4f70e8b27143b698a1d68f2a04b9bd799d41842f1eec76411d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:23:43.331 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:42.972+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:43.332 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:42 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:42.856+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:43.602 INFO:journalctl@ceph.osd.3.smithi002.stdout:Feb 13 15:23:43 smithi002 podman[92281]: 2024-02-13 15:23:43.32338327 +0000 UTC m=+0.700281587 container died 01e1196601b6be4f70e8b27143b698a1d68f2a04b9bd799d41842f1eec76411d (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:43.832 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:43.536+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:43.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:43.361+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:43.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.3.service' 2024-02-13T15:23:43.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:43.932 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-13T15:23:43.932 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-13T15:23:43.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.4 2024-02-13T15:23:44.238 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:43.860+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:44.238 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:43 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:43.936+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:44.238 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:43 smithi098 systemd[1]: Stopping Ceph osd.4 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:44.546 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:44.377+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:44.547 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:44.236+0000 7f781e80e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:44.547 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:44.236+0000 7f781e80e700 -1 osd.4 151 *** Got signal Terminated *** 2024-02-13T15:23:44.547 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:44.236+0000 7f781e80e700 -1 osd.4 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:44.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:23:44.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:44.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:44.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:44.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:44 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:44.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:44.825 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:44.545+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:45.081 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:44.922+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:45.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:44 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:44.824+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:45.801 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:45.538+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:45.801 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:45.426+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:46.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:45.800+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:46.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:45 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:45.900+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:46.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:46.430+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:46.832 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:46.501+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:47.332 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:46.944+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:47.332 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:46 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:46.834+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:47.811 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:47.519+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:47.811 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:47.416+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:48.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:47.809+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:48.082 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:47 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:47.992+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:48.779 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:48.551+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:48.780 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:48.551+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:48.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:48.408+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:48.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:48.408+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:49.081 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4[61327]: 2024-02-13T15:23:48.998+0000 7f781ae02700 -1 osd.4 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:12.173480+0000 front 2024-02-13T15:23:12.173571+0000 (oldest deadline 2024-02-13T15:23:42.173750+0000) 2024-02-13T15:23:49.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:48 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:48.778+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:49.779 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:49.600+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:49.780 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:49.600+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:49.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:49.408+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:49.780 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:49.408+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:50.081 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:49 smithi098 podman[76812]: 2024-02-13 15:23:49.779007962 +0000 UTC m=+5.648025820 container died 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:50.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:49.796+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:50.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:49 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:49.796+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:50.448 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:50 smithi098 podman[76812]: 2024-02-13 15:23:50.237424138 +0000 UTC m=+6.106442047 container cleanup 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:23:50.449 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:50 smithi098 bash[76812]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4 2024-02-13T15:23:50.807 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:50 smithi098 podman[76829]: 2024-02-13 15:23:50.497180054 +0000 UTC m=+0.716653473 container remove 185eb10048c41d26fb0d22329f6ec026dd573020394f1a345d930bc915b1f5f0 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-13T15:23:50.807 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:50.600+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:50.807 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:50.600+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:50.807 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:50.447+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:50.807 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:50.447+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:51.076 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:50 smithi098 podman[76920]: 2024-02-13 15:23:50.709144969 +0000 UTC m=+0.018649438 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:51.077 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:50 smithi098 podman[76920]: 2024-02-13 15:23:50.806681321 +0000 UTC m=+0.116185821 container create 31429ae9045301b1fb09c7e74614d195615fd88d78a02c7e27dc79962643a39e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-13T15:23:51.077 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:50.834+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:51.077 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:50 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:50.834+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:51.332 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:51 smithi098 podman[76920]: 2024-02-13 15:23:51.076179519 +0000 UTC m=+0.385683980 container init 31429ae9045301b1fb09c7e74614d195615fd88d78a02c7e27dc79962643a39e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-13T15:23:51.332 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:51 smithi098 podman[76920]: 2024-02-13 15:23:51.080027323 +0000 UTC m=+0.389531775 container start 31429ae9045301b1fb09c7e74614d195615fd88d78a02c7e27dc79962643a39e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:51.332 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:51 smithi098 podman[76920]: 2024-02-13 15:23:51.167994789 +0000 UTC m=+0.477499259 container attach 31429ae9045301b1fb09c7e74614d195615fd88d78a02c7e27dc79962643a39e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:23:51.645 INFO:journalctl@ceph.osd.4.smithi098.stdout:Feb 13 15:23:51 smithi098 podman[76920]: 2024-02-13 15:23:51.379677976 +0000 UTC m=+0.689182442 container died 31429ae9045301b1fb09c7e74614d195615fd88d78a02c7e27dc79962643a39e (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:51.645 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:51.496+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:51.645 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:51.496+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:51.922 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:51.643+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:51.922 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:51.643+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:51.922 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:51.858+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:51.922 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:51 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:51.858+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:51.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.4.service' 2024-02-13T15:23:51.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:23:51.981 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-13T15:23:51.981 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-13T15:23:51.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.5 2024-02-13T15:23:52.252 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 systemd[1]: Stopping Ceph osd.5 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:23:52.517 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:52.358+0000 7fbb9c889700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:23:52.517 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:52.358+0000 7fbb9c889700 -1 osd.5 151 *** Got signal Terminated *** 2024-02-13T15:23:52.517 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:52.358+0000 7fbb9c889700 -1 osd.5 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:23:52.816 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:52.675+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:52.816 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:52.675+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:52.816 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:52.516+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:52.816 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:52.516+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:53.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:52.815+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:53.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:52 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:52.815+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:53.817 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:53.703+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:53.817 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:53.703+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:53.817 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:53.485+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:53.817 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:53.485+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:53.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:53.539Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:53.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:53 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:53.539Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:54.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:53.815+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:54.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:53 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:53.815+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:54.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:23:54.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:54.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:54.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:23:54.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:23:54 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:23:54.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:23:54.785 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:54.655+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:54.785 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:54.655+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:54.785 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:54.467+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:54.785 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:54.467+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:55.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:54.784+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:55.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:54 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:54.784+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:55.768 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:55.654+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:55.768 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:55.654+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:55.768 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:55.506+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:55.768 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:55.506+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:56.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:55.767+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:56.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:55.767+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:56.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:55 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:55.767+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:23:56.789 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:56.526+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:56.789 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:56.526+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:56.789 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:56.669+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:16.055300+0000 front 2024-02-13T15:23:16.055158+0000 (oldest deadline 2024-02-13T15:23:38.955159+0000) 2024-02-13T15:23:56.789 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:56.669+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.156718+0000 front 2024-02-13T15:23:24.156564+0000 (oldest deadline 2024-02-13T15:23:47.656518+0000) 2024-02-13T15:23:56.789 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5[63654]: 2024-02-13T15:23:56.669+0000 7fbb94e75700 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.357721+0000 front 2024-02-13T15:23:32.357609+0000 (oldest deadline 2024-02-13T15:23:56.457514+0000) 2024-02-13T15:23:57.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:56.788+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:57.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:56.788+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:57.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:56 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:56.788+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:23:57.785 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:57.519+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:57.785 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:57.519+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:58.081 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:57 smithi098 podman[77036]: 2024-02-13 15:23:57.85004805 +0000 UTC m=+5.642250967 container died edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-13T15:23:58.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:57.782+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:58.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:57.782+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:58.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:57.782+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:23:58.478 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:58 smithi098 podman[77036]: 2024-02-13 15:23:58.276077657 +0000 UTC m=+6.068280618 container cleanup edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:23:58.478 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:58 smithi098 bash[77036]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5 2024-02-13T15:23:58.768 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:58 smithi098 podman[77054]: 2024-02-13 15:23:58.477358675 +0000 UTC m=+0.624342683 container remove edf5d8026920eb6ec4c0ac01c4a6f89bbe29511588c997964fd07507d1c5fb95 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-13T15:23:58.768 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:58.486+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:58.768 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:58.486+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:58.768 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:58.486+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:23:59.034 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:58 smithi098 podman[77149]: 2024-02-13 15:23:58.801885548 +0000 UTC m=+0.018265700 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:23:59.035 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:58 smithi098 podman[77149]: 2024-02-13 15:23:58.903322244 +0000 UTC m=+0.119702383 container create c4e5c3fc32f0526ab1b3570a288555848b1dd2b7e216ad729d7007d2c0b6331f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-13T15:23:59.035 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:58.766+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:23:59.035 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:58.766+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:23:59.035 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:58 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:58.766+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:23:59.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:59 smithi098 podman[77149]: 2024-02-13 15:23:59.173484098 +0000 UTC m=+0.389864248 container init c4e5c3fc32f0526ab1b3570a288555848b1dd2b7e216ad729d7007d2c0b6331f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-13T15:23:59.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:59 smithi098 podman[77149]: 2024-02-13 15:23:59.17704744 +0000 UTC m=+0.393427584 container start c4e5c3fc32f0526ab1b3570a288555848b1dd2b7e216ad729d7007d2c0b6331f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-13T15:23:59.473 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:59 smithi098 podman[77149]: 2024-02-13 15:23:59.256243391 +0000 UTC m=+0.472623542 container attach c4e5c3fc32f0526ab1b3570a288555848b1dd2b7e216ad729d7007d2c0b6331f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True) 2024-02-13T15:23:59.788 INFO:journalctl@ceph.osd.5.smithi098.stdout:Feb 13 15:23:59 smithi098 podman[77149]: 2024-02-13 15:23:59.473318279 +0000 UTC m=+0.689698430 container died c4e5c3fc32f0526ab1b3570a288555848b1dd2b7e216ad729d7007d2c0b6331f (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:23:59.788 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:59.493+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:23:59.788 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:59.493+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:23:59.788 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:23:59.493+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:00.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:59.787+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:00.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:59.787+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:00.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:23:59 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:23:59.787+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:00.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.5.service' 2024-02-13T15:24:00.252 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:24:00.252 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-13T15:24:00.252 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-13T15:24:00.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.6 2024-02-13T15:24:00.497 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 systemd[1]: Stopping Ceph osd.6 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.495+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.495+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.495+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.604+0000 7f8010faf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.604+0000 7f8010faf700 -1 osd.6 151 *** Got signal Terminated *** 2024-02-13T15:24:00.764 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:00.604+0000 7f8010faf700 -1 osd.6 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:24:01.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:00.762+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:01.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:00.762+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:01.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:00 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:00.762+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:01.542+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:01.542+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:01.542+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:01.730+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:01.730+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:01.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:01 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:01.730+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:02.581 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:02.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:02.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:02.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:02.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:02.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:02.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:02.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:37.657790+0000 front 2024-02-13T15:23:37.657801+0000 (oldest deadline 2024-02-13T15:24:02.357761+0000) 2024-02-13T15:24:03.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:02.684+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:03.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:02.684+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:03.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:02 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:02.684+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:03.719+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:03.719+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:03.719+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:03.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:03.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:03.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:03.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:03 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:03.504+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:37.657790+0000 front 2024-02-13T15:23:37.657801+0000 (oldest deadline 2024-02-13T15:24:02.357761+0000) 2024-02-13T15:24:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:24:03.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:03.540Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:03.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:03 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:03.540Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:04.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:24:04.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:04.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:04.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:04.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:04 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:04.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:04.831 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:04.690+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:04.690+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:04.690+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:04.456+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:04.456+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:04.456+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:04.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:04 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:04.456+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:37.657790+0000 front 2024-02-13T15:23:37.657801+0000 (oldest deadline 2024-02-13T15:24:02.357761+0000) 2024-02-13T15:24:05.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:05.505+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:11.954714+0000 front 2024-02-13T15:23:11.954635+0000 (oldest deadline 2024-02-13T15:23:37.254535+0000) 2024-02-13T15:24:05.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:05.505+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:23.655862+0000 front 2024-02-13T15:23:23.656231+0000 (oldest deadline 2024-02-13T15:23:47.755862+0000) 2024-02-13T15:24:05.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:05.505+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:32.957364+0000 front 2024-02-13T15:23:32.957303+0000 (oldest deadline 2024-02-13T15:23:57.657308+0000) 2024-02-13T15:24:05.832 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6[65980]: 2024-02-13T15:24:05.505+0000 7f800959b700 -1 osd.6 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:37.657790+0000 front 2024-02-13T15:23:37.657801+0000 (oldest deadline 2024-02-13T15:24:02.357761+0000) 2024-02-13T15:24:05.832 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:05.670+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:05.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:05.670+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:05.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:05.670+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:05.833 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:05 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:05.670+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:06.206 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:05 smithi098 podman[77254]: 2024-02-13 15:24:05.931638692 +0000 UTC m=+5.461149378 container died 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:24:06.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:06 smithi098 podman[77254]: 2024-02-13 15:24:06.380653944 +0000 UTC m=+5.910164627 container cleanup 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-13T15:24:06.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:06 smithi098 bash[77254]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6 2024-02-13T15:24:06.997 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:06 smithi098 podman[77272]: 2024-02-13 15:24:06.582020513 +0000 UTC m=+0.643637186 container remove 9608c796f9a7bd16e84dac329ea9f9e3081f64a271f929a6a18bacb64867f4fc (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:24:06.997 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:06.638+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:06.997 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:06.638+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:06.997 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:06.638+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:06.997 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:06 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:06.638+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:07.286 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:06 smithi098 podman[77366]: 2024-02-13 15:24:06.899385397 +0000 UTC m=+0.017511057 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:24:07.286 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:07 smithi098 podman[77366]: 2024-02-13 15:24:07.036775003 +0000 UTC m=+0.154900689 container create b02e65cd35026faf71f6d78e543ceea20f59cfa10d3bcde7370e058f7de7b62c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:24:07.581 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:07 smithi098 podman[77366]: 2024-02-13 15:24:07.285843907 +0000 UTC m=+0.403969563 container init b02e65cd35026faf71f6d78e543ceea20f59cfa10d3bcde7370e058f7de7b62c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T15:24:07.581 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:07 smithi098 podman[77366]: 2024-02-13 15:24:07.289006658 +0000 UTC m=+0.407132311 container start b02e65cd35026faf71f6d78e543ceea20f59cfa10d3bcde7370e058f7de7b62c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-13T15:24:07.582 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:07 smithi098 podman[77366]: 2024-02-13 15:24:07.369242188 +0000 UTC m=+0.487367864 container attach b02e65cd35026faf71f6d78e543ceea20f59cfa10d3bcde7370e058f7de7b62c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-13T15:24:07.873 INFO:journalctl@ceph.osd.6.smithi098.stdout:Feb 13 15:24:07 smithi098 podman[77366]: 2024-02-13 15:24:07.582689784 +0000 UTC m=+0.700815439 container died b02e65cd35026faf71f6d78e543ceea20f59cfa10d3bcde7370e058f7de7b62c (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-13T15:24:07.873 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:07.681+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:07.873 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:07.681+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:07.873 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:07.681+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:07.873 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:07 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:07.681+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:08.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.6.service' 2024-02-13T15:24:08.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:24:08.242 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-13T15:24:08.243 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-13T15:24:08.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.7 2024-02-13T15:24:08.440 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 systemd[1]: Stopping Ceph osd.7 for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:08.709 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.592+0000 7f177579c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T15:24:08.709 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.592+0000 7f177579c700 -1 osd.7 151 *** Got signal Terminated *** 2024-02-13T15:24:08.710 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.592+0000 7f177579c700 -1 osd.7 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T15:24:09.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.708+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:09.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.708+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:09.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.708+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:09.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:08 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:08.708+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:10.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:09.679+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:10.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:09.679+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:10.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:09.679+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:10.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:09 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:09.679+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:11.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:10.658+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:11.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:10.658+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:11.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:10.658+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:11.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:10 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:10.658+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:12.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:11 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:11.621+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:12.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:11 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:11.621+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:12.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:11 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:11.621+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:12.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:11 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:11.621+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:13.081 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:12.616+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-02-13T15:23:13.774403+0000 front 2024-02-13T15:23:13.774333+0000 (oldest deadline 2024-02-13T15:23:37.274275+0000) 2024-02-13T15:24:13.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:12.616+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-02-13T15:23:24.876063+0000 front 2024-02-13T15:23:24.876142+0000 (oldest deadline 2024-02-13T15:23:49.576016+0000) 2024-02-13T15:24:13.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:12.616+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-02-13T15:23:33.076859+0000 front 2024-02-13T15:23:33.076878+0000 (oldest deadline 2024-02-13T15:23:55.376791+0000) 2024-02-13T15:24:13.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:12.616+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-02-13T15:23:39.477878+0000 front 2024-02-13T15:23:39.477915+0000 (oldest deadline 2024-02-13T15:24:05.377825+0000) 2024-02-13T15:24:13.082 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:12 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7[68336]: 2024-02-13T15:24:12.616+0000 7f176dd88700 -1 osd.7 151 heartbeat_check: no reply from 172.21.15.98:6804 osd.4 since back 2024-02-13T15:23:47.078712+0000 front 2024-02-13T15:23:47.078695+0000 (oldest deadline 2024-02-13T15:24:11.778649+0000) 2024-02-13T15:24:13.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:24:13.539Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:13.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:13.540Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:13.876 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:13 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:13.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:14.310 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:13 smithi098 podman[77473]: 2024-02-13 15:24:13.935250451 +0000 UTC m=+5.461998829 container died 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T15:24:14.310 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:14 smithi098 podman[77473]: 2024-02-13 15:24:14.309627385 +0000 UTC m=+5.836375808 container cleanup 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:24:14.581 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:14 smithi098 bash[77473]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7 2024-02-13T15:24:14.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=error ts=2024-02-13T15:24:14.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:14.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-02-13T15:24:14.626 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:14 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=warn ts=2024-02-13T15:24:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-02-13T15:24:14.926 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:14 smithi098 podman[77502]: 2024-02-13 15:24:14.586178488 +0000 UTC m=+0.651608171 container remove 00de955572baa43688d3262b2a10b92084c503166581a842432309c38a9018b1 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T15:24:15.240 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:14 smithi098 podman[77595]: 2024-02-13 15:24:14.829055678 +0000 UTC m=+0.017879579 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T15:24:15.240 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:14 smithi098 podman[77595]: 2024-02-13 15:24:14.92869155 +0000 UTC m=+0.117515437 container create 63af7d43678d5e63f093d10a274bf90263d04212e7378808998ff25f791e7396 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:24:15.240 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:15 smithi098 podman[77595]: 2024-02-13 15:24:15.156897686 +0000 UTC m=+0.345721582 container init 63af7d43678d5e63f093d10a274bf90263d04212e7378808998ff25f791e7396 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-13T15:24:15.240 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:15 smithi098 podman[77595]: 2024-02-13 15:24:15.160181575 +0000 UTC m=+0.349005462 container start 63af7d43678d5e63f093d10a274bf90263d04212e7378808998ff25f791e7396 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:24:15.241 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:15 smithi098 podman[77595]: 2024-02-13 15:24:15.239828644 +0000 UTC m=+0.428652545 container attach 63af7d43678d5e63f093d10a274bf90263d04212e7378808998ff25f791e7396 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T15:24:15.581 INFO:journalctl@ceph.osd.7.smithi098.stdout:Feb 13 15:24:15 smithi098 podman[77595]: 2024-02-13 15:24:15.449242636 +0000 UTC m=+0.638066527 container died 63af7d43678d5e63f093d10a274bf90263d04212e7378808998ff25f791e7396 (image=quay.ceph.io/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T15:24:16.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@osd.7.service' 2024-02-13T15:24:16.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:24:16.096 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-13T15:24:16.097 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-13T15:24:16.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a 2024-02-13T15:24:16.284 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 systemd[1]: Stopping Ceph prometheus.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:16.574 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.280Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:24:16.574 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-13T15:24:16.574 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-13T15:24:16.574 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-13T15:24:16.574 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a[53930]: ts=2024-02-13T15:24:16.281Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-13T15:24:16.575 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 podman[77674]: 2024-02-13 15:24:16.291378306 +0000 UTC m=+0.104130599 container died 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:16.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 podman[77674]: 2024-02-13 15:24:16.739769597 +0000 UTC m=+0.552521899 container cleanup 08bdc7bdc6b44ed8e4dd85a45833c37b2b5eabd8608a890082fdf9ac5f7aed49 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:16.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 bash[77674]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T15:24:16.832 INFO:journalctl@ceph.prometheus.a.smithi098.stdout:Feb 13 15:24:16 smithi098 bash[77700]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T15:24:17.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-13T15:24:17.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T15:24:17.066 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-13T15:24:17.066 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T15:24:18.876 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:18 smithi002 systemd[1]: Stopping Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:18.876 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:18 smithi002 podman[92646]: 2024-02-13 15:24:18.714133086 +0000 UTC m=+0.122586995 container died 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:19.330 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 podman[92646]: 2024-02-13 15:24:19.078356119 +0000 UTC m=+0.486810013 container cleanup 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:19.330 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 bash[92646]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a 2024-02-13T15:24:19.330 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 bash[92673]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-13T15:24:19.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 podman[92660]: 2024-02-13 15:24:19.3295327 +0000 UTC m=+0.618561547 container remove 9e47fb0e679615635e197e70e594874e95658cb2f18684203d9c096dffe56fdf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:19.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:24:19.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-13T15:24:19.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 systemd[1]: Stopped Ceph node-exporter.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:24:19.626 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Feb 13 15:24:19 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 4.032s CPU time. 2024-02-13T15:24:19.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:19 smithi002 systemd[1]: Stopping Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:19.950 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:19 smithi002 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a[58778]: level=info ts=2024-02-13T15:24:19.640Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-13T15:24:19.950 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:19 smithi002 podman[92726]: 2024-02-13 15:24:19.681701717 +0000 UTC m=+0.143239214 container died a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:20.309 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 podman[92726]: 2024-02-13 15:24:20.065995548 +0000 UTC m=+0.527533052 container cleanup a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:20.309 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 bash[92726]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T15:24:20.309 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 bash[92754]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T15:24:20.566 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 podman[92740]: 2024-02-13 15:24:20.308731082 +0000 UTC m=+0.622129696 container remove a9e94646913c4bc5102346ad42a2f0f4956fb9668338299b3551133b8a8528cf (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T15:24:20.566 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 podman[92740]: 2024-02-13 15:24:20.384215268 +0000 UTC m=+0.697613885 volume remove d1abf9e8a5d45a2244f72e6a708ccdb275bef79f10e5060dac2ebd3d8f16be55 2024-02-13T15:24:20.566 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-13T15:24:20.566 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 systemd[1]: Stopped Ceph alertmanager.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:24:20.567 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Feb 13 15:24:20 smithi002 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@alertmanager.a.service: Consumed 1.059s CPU time. 2024-02-13T15:24:54.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T15:24:56.328 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 systemd[1]: Stopping Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:56.581 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 podman[78046]: 2024-02-13 15:24:56.326302757 +0000 UTC m=+0.112999642 container died 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:24:56.935 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 podman[78046]: 2024-02-13 15:24:56.757889851 +0000 UTC m=+0.544586734 container cleanup 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:24:56.935 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 bash[78046]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b 2024-02-13T15:24:57.255 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 systemd[1]: Stopping Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715... 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 podman[78060]: 2024-02-13 15:24:56.934512014 +0000 UTC m=+0.610491510 container remove 1735010684f0ebc88b97bcb262b93830f2e89c6ef86713f8d7a820b16294f147 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 bash[78076]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 systemd[1]: Stopped Ceph node-exporter.b for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:24:57.255 INFO:journalctl@ceph.node-exporter.b.smithi098.stdout:Feb 13 15:24:56 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 3.799s CPU time. 2024-02-13T15:24:57.538 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a[51070]: t=2024-02-13T15:24:57+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-13T15:24:57.539 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 podman[78125]: 2024-02-13 15:24:57.265616661 +0000 UTC m=+0.117428678 container died f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.24.2, io.openshift.expose-services=, release=236.1648460182, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container) 2024-02-13T15:24:57.831 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 podman[78125]: 2024-02-13 15:24:57.612411217 +0000 UTC m=+0.464223239 container cleanup f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-type=git, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5) 2024-02-13T15:24:57.831 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 bash[78125]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a 2024-02-13T15:24:57.832 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 bash[78152]: Error: no container with name or ID "ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-13T15:24:58.301 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 podman[78140]: 2024-02-13 15:24:57.880273381 +0000 UTC m=+0.618045627 container remove f31dbe3f643adad3204594328049bde7b3ea9f3ecd271faf9385ac79dcff24ef (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c0fca16a-ca80-11ee-95b9-87774f69a715-grafana-a, distribution-scope=public, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., vcs-type=git, maintainer=Paul Cuzner , release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-13T15:24:58.301 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-13T15:24:58.301 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 systemd[1]: Stopped Ceph grafana.a for c0fca16a-ca80-11ee-95b9-87774f69a715. 2024-02-13T15:24:58.301 INFO:journalctl@ceph.grafana.a.smithi098.stdout:Feb 13 15:24:57 smithi098 systemd[1]: ceph-c0fca16a-ca80-11ee-95b9-87774f69a715@grafana.a.service: Consumed 4.113s CPU time. 2024-02-13T15:25:20.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:25:20.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T15:25:20.969 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-13T15:25:20.971 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi002/crash 2024-02-13T15:25:20.972 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash -- . 2024-02-13T15:25:21.015 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T15:25:21.015 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2024-02-13T15:25:21.017 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi098/crash 2024-02-13T15:25:21.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash -- . 2024-02-13T15:25:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T15:25:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-02-13T15:25:21.061 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-13T15:25:21.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T15:25:21.092 INFO:teuthology.orchestra.run.smithi002.stdout:2024-02-13T15:03:30.599385+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T15:25:21.092 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-13T15:25:21.093 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[SEC\]' /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T15:25:21.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]' /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T15:25:21.233 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[WRN\]' /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T15:25:21.301 INFO:teuthology.orchestra.run.smithi002.stdout:2024-02-13T15:03:30.599385+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T15:25:21.302 INFO:tasks.cephadm:Compressing logs... 2024-02-13T15:25:21.302 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T15:25:21.344 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T15:25:21.371 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-13T15:25:21.371 INFO:teuthology.orchestra.run.smithi002.stderr:: No such file or directory 2024-02-13T15:25:21.371 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi002.jwgbut.log 2024-02-13T15:25:21.371 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi002.szwgvn.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.a.log: /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi002.jwgbut.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi002.szwgvn.log: 76.7% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi002.jwgbut.log.gz 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr: 76.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi002.szwgvn.log.gz 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.3.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log: /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/tcmu-runner.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr: 84.4% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/tcmu-runner.log.gz 2024-02-13T15:25:21.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log 2024-02-13T15:25:21.374 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log: 91.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T15:25:21.374 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.6.log 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi098.eccznt.log 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.4.log 2024-02-13T15:25:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi098.eccznt.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.7.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr: 76.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi098.eccznt.log.gz 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.5.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.4.log: /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log: /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log: gzip -5 --verbose 85.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log: 90.4% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi098.kzxzji.log 2024-02-13T15:25:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-13T15:25:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi098.kzxzji.log: 76.3% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-client.rgw.smpl.smithi098.kzxzji.log.gz 2024-02-13T15:25:21.383 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log: 94.2% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T15:25:21.383 INFO:teuthology.orchestra.run.smithi002.stderr: 93.0% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T15:25:21.383 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T15:25:21.383 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-13T15:25:21.388 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: 91.4% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T15:25:21.389 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T15:25:21.391 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-13T15:25:21.391 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.c.log: 91.2% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T15:25:21.416 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T15:25:21.432 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-13T15:25:21.739 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-13T15:25:21.820 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-13T15:25:21.974 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-13T15:25:22.383 INFO:teuthology.orchestra.run.smithi002.stderr: 91.0% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-13T15:25:22.537 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-13T15:25:22.593 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.6.log.gz 2024-02-13T15:25:22.653 INFO:teuthology.orchestra.run.smithi098.stderr: 93.8% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.7.log.gz 2024-02-13T15:25:22.667 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.5.log.gz 2024-02-13T15:25:22.847 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.4.log.gz 2024-02-13T15:25:22.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-02-13T15:25:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.490s 2024-02-13T15:25:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m5.593s 2024-02-13T15:25:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.237s 2024-02-13T15:25:22.976 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-13T15:25:23.060 INFO:teuthology.orchestra.run.smithi002.stderr: 93.7% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-13T15:25:23.302 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/c0fca16a-ca80-11ee-95b9-87774f69a715/ceph-osd.3.log.gz 2024-02-13T15:25:23.304 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-13T15:25:23.304 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m1.948s 2024-02-13T15:25:23.304 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m8.099s 2024-02-13T15:25:23.304 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.261s 2024-02-13T15:25:23.304 INFO:tasks.cephadm:Archiving logs... 2024-02-13T15:25:23.304 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi002/log 2024-02-13T15:25:23.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T15:25:24.253 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi098/log 2024-02-13T15:25:24.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T15:25:25.032 INFO:tasks.cephadm:Removing cluster... 2024-02-13T15:25:25.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 --force 2024-02-13T15:25:25.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0fca16a-ca80-11ee-95b9-87774f69a715 --force 2024-02-13T15:25:25.614 INFO:tasks.cephadm:Removing cephadm ... 2024-02-13T15:25:25.614 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T15:25:25.629 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T15:25:25.643 INFO:tasks.cephadm:Teardown complete 2024-02-13T15:25:25.643 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-13T15:25:25.652 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-13T15:25:25.653 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T15:25:25.672 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T15:25:25.683 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -129us[ -129us] +/- 81ms 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -1593us[-1855us] +/- 49ms 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +2145us[+2145us] +/- 71ms 2024-02-13T15:25:25.687 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:25:25.696 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-02-13T15:25:25.700 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T15:25:25.700 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-02-13T15:25:25.701 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -78us[ -92us] +/- 81ms 2024-02-13T15:25:25.701 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -1534us[-1549us] +/- 49ms 2024-02-13T15:25:25.701 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +2207us[+2192us] +/- 71ms 2024-02-13T15:25:25.701 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T15:25:25.701 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-13T15:25:25.710 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-13T15:25:25.710 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-13T15:25:25.719 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T15:25:25.810 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T15:25:25.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T15:25:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T15:25:25.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T15:25:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T15:25:25.946 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-13T15:25:25.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-13T15:25:25.996 INFO:teuthology.task.internal:Duration was 1700.567241 seconds 2024-02-13T15:25:25.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-13T15:25:26.004 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-13T15:25:26.005 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T15:25:26.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T15:25:26.048 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T15:25:26.050 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T15:25:26.413 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-13T15:25:26.413 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-02-13T15:25:26.413 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T15:25:26.473 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-02-13T15:25:26.473 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T15:25:26.494 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-13T15:25:26.494 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T15:25:26.516 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T15:25:26.562 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-13T15:25:26.562 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T15:25:26.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T15:25:27.011 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-13T15:25:27.021 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-13T15:25:27.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T15:25:27.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T15:25:27.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-13T15:25:27.086 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T15:25:27.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T15:25:27.114 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-02-13T15:25:27.143 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-02-13T15:25:27.159 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T15:25:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T15:25:27.202 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T15:25:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T15:25:27.216 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-13T15:25:27.226 INFO:teuthology.task.internal:Transferring archived files... 2024-02-13T15:25:27.227 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi002 2024-02-13T15:25:27.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T15:25:27.290 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098/remote/smithi098 2024-02-13T15:25:27.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T15:25:27.332 INFO:teuthology.task.internal:Removing archive directory... 2024-02-13T15:25:27.332 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T15:25:27.335 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T15:25:27.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-13T15:25:27.391 INFO:teuthology.task.internal:Not uploading archives. 2024-02-13T15:25:27.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-13T15:25:27.401 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-13T15:25:27.402 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T15:25:27.404 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T15:25:27.416 INFO:teuthology.orchestra.run.smithi002.stdout: 265807 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 15:25 /home/ubuntu/cephtest 2024-02-13T15:25:27.433 INFO:teuthology.orchestra.run.smithi098.stdout: 265984 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 15:25 /home/ubuntu/cephtest 2024-02-13T15:25:27.434 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-13T15:25:27.444 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-13T15:25:27.468 INFO:teuthology.nuke:Checking targets against current locks 2024-02-13T15:25:27.495 DEBUG:teuthology.nuke:shortname: smithi002 2024-02-13T15:25:27.495 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T15:25:27.518 DEBUG:teuthology.nuke:shortname: smithi098 2024-02-13T15:25:27.519 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T15:25:27.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:48:57.730978', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPAoKKIJhqZ3r5YochdFAD9JRIWbhb0onyuXid4rRVyBCSwGUDpdSuMAroxNUke3UW6SMTgpdJlt0QFrArmlvc='} 2024-02-13T15:25:27.542 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:48:57.729610', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIUjl3gaY9DGwFloYSKxb6NtjiX9PjiiG4Bz2Ehh0H1cc2C2oMtZZsKe6wXj4ec03EF9kN+bGZbKmfXlgVXXmrY='} 2024-02-13T15:25:27.576 INFO:teuthology.orchestra.console.smithi002:Power off 2024-02-13T15:25:27.576 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T15:25:27.599 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2024-02-13T15:25:27.600 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:27.613 INFO:teuthology.orchestra.console.smithi098:Power off 2024-02-13T15:25:27.613 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T15:25:27.623 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-02-13T15:25:27.635 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-02-13T15:25:27.635 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:27.658 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-02-13T15:25:31.624 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:31.659 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:31.733 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-02-13T15:25:31.767 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-02-13T15:25:35.738 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:35.768 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:35.852 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2024-02-13T15:25:35.877 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-02-13T15:25:35.954 INFO:teuthology.orchestra.console.smithi002:Power off completed 2024-02-13T15:25:39.882 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T15:25:39.991 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-02-13T15:25:40.092 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-02-13T15:25:40.195 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1700.5672407150269 failure_reason: '"2024-02-13T15:03:30.599385+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_phlogistonjohn@teuthology success: false 2024-02-13T15:25:40.196 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T15:25:40.268 INFO:teuthology.run:FAIL