2024-02-27T07:30:01.327 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-27T07:30:01.327 DEBUG:teuthology.run:Teuthology command: teuthology --name adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi --archive /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539 --verbose --owner scheduled_adking@teuthology --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} -- /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/orig.config.yaml 2024-02-27T07:30:01.356 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-27T07:30:01.438 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539 branch: wip-adk-testing-2024-02-26-1600 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: adking@redhat.com first_in_suite: false job_id: '7574539' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-02-26-1600 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN - PG_DEGRADED - PG_AVAILABILITY sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 workunit: branch: wip-adk-testing-2024-02-26-1600 sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3790 sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-adk-testing-2024-02-26-1600 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 targets: smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-27_05:08:20 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-27T07:30:01.438 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa; will attempt to use it 2024-02-27T07:30:01.439 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa/tasks 2024-02-27T07:30:01.439 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-27T07:30:01.441 INFO:teuthology.task.internal:Checking packages... 2024-02-27T07:30:01.462 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '635ee6850c57c98cc6ba932cade45f9cc015aa32' 2024-02-27T07:30:01.462 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-27T07:30:01.463 INFO:teuthology.packaging:ref: None 2024-02-27T07:30:01.463 INFO:teuthology.packaging:tag: None 2024-02-27T07:30:01.463 INFO:teuthology.packaging:branch: wip-adk-testing-2024-02-26-1600 2024-02-27T07:30:01.463 INFO:teuthology.packaging:sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:30:01.463 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-02-26-1600 2024-02-27T07:30:01.620 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1736.g635ee685 2024-02-27T07:30:01.621 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-27T07:30:01.628 INFO:teuthology.task.internal:no buildpackages task found 2024-02-27T07:30:01.629 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-27T07:30:01.665 INFO:teuthology.task.internal:Saving configuration 2024-02-27T07:30:01.676 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-27T07:30:01.698 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T07:30:01.720 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 07:24:53.398304', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo='} 2024-02-27T07:30:01.745 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 07:24:53.400082', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='} 2024-02-27T07:30:01.745 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-27T07:30:01.752 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-27T07:30:01.752 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-27T07:30:01.752 INFO:teuthology.run_tasks:Running task console_log... 2024-02-27T07:30:01.848 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efdf3db9160>, signals=[15]) 2024-02-27T07:30:01.848 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-27T07:30:01.855 INFO:teuthology.task.internal:Opening connections... 2024-02-27T07:30:01.855 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T07:30:01.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:30:01.948 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-02-27T07:30:01.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:30:02.036 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-27T07:30:02.043 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-02-27T07:30:02.062 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-02-27T07:30:02.062 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="9" 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="9" 2024-02-27T07:30:02.119 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el9" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:LOGO="fedora-logo-icon" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-27T07:30:02.120 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-27T07:30:02.121 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-02-27T07:30:02.142 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-02-27T07:30:02.158 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-02-27T07:30:02.159 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-27T07:30:02.215 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-27T07:30:02.216 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-27T07:30:02.216 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-02-27T07:30:02.242 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-27T07:30:02.252 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-27T07:30:02.283 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-27T07:30:02.283 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-02-27T07:30:02.286 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-02-27T07:30:02.301 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-27T07:30:02.316 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-27T07:30:02.316 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-02-27T07:30:02.343 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-02-27T07:30:02.366 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-27T07:30:02.419 INFO:teuthology.run_tasks:Running task kernel... 2024-02-27T07:30:02.434 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-27T07:30:02.434 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-27T07:30:02.434 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-27T07:30:02.435 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-27T07:30:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T07:30:02.450 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-27T07:30:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T07:30:02.451 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-02-27T07:30:02.506 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-361.el9.x86_64 2024-02-27T07:30:02.506 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-361.el9.x86_64 2024-02-27T07:30:02.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-02-27T07:30:02.507 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-361.el9.x86_64 2024-02-27T07:30:02.508 INFO:teuthology.task.kernel:Running kernel on smithi149: 5.14.0-361.el9.x86_64 2024-02-27T07:30:02.508 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-02-27T07:30:04.835 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 11 MB/s | 8.1 MB 00:00 2024-02-27T07:30:06.851 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.5 MB 00:00 2024-02-27T07:30:07.056 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - AppStream 2.8 MB/s | 8.1 MB 00:02 2024-02-27T07:30:07.825 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-02-27T07:30:08.680 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - BaseOS 8.4 MB/s | 2.5 MB 00:00 2024-02-27T07:30:09.898 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-27T07:30:12.183 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-02-27T07:30:17.264 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-02-27T07:30:19.098 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-02-27T07:30:20.252 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:00 2024-02-27T07:30:25.357 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-27T07:30:27.618 INFO:teuthology.orchestra.run.smithi149.stderr:Modular dependency problems: 2024-02-27T07:30:27.618 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 1: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 2: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 3: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 4: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 5: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 6: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 7: conflicting requests 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.619 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 8: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 9: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 10: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 11: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 12: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 13: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 14: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 15: conflicting requests 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.620 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 16: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 17: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 18: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 19: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 20: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 21: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 22: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 23: conflicting requests 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.621 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 24: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 25: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 26: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 27: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 28: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 29: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 30: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 31: conflicting requests 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.622 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 32: conflicting requests 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 33: conflicting requests 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 34: conflicting requests 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 35: conflicting requests 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 36: conflicting requests 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 37: cannot install the best candidate for the job 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.623 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 38: conflicting requests 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 39: cannot install the best candidate for the job 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 40: cannot install the best candidate for the job 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.624 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 41: cannot install the best candidate for the job 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 42: cannot install the best candidate for the job 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 43: cannot install the best candidate for the job 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.625 INFO:teuthology.orchestra.run.smithi149.stderr: Problem 44: cannot install the best candidate for the job 2024-02-27T07:30:27.626 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.626 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.626 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.626 INFO:teuthology.orchestra.run.smithi149.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-27T07:30:27.973 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-27T07:30:27.974 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-27T07:30:28.084 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 5.14.0-419.el9 baseos 6.0 M 2024-02-27T07:30:28.091 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core x86_64 5.14.0-419.el9 baseos 21 M 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules x86_64 5.14.0-419.el9 baseos 39 M 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core x86_64 5.14.0-419.el9 baseos 33 M 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout:Install 4 Packages 2024-02-27T07:30:28.092 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:30:28.093 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 99 M 2024-02-27T07:30:28.093 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 122 M 2024-02-27T07:30:28.093 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-27T07:30:29.034 INFO:teuthology.orchestra.run.smithi149.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 7.5 MB/s | 6.0 MB 00:00 2024-02-27T07:30:29.823 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 937 kB/s | 19 MB 00:20 2024-02-27T07:30:30.901 INFO:teuthology.orchestra.run.smithi149.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 7.8 MB/s | 21 MB 00:02 2024-02-27T07:30:32.451 INFO:teuthology.orchestra.run.smithi149.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 9.8 MB/s | 33 MB 00:03 2024-02-27T07:30:33.576 INFO:teuthology.orchestra.run.smithi149.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 7.3 MB/s | 39 MB 00:05 2024-02-27T07:30:33.576 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-27T07:30:33.577 INFO:teuthology.orchestra.run.smithi149.stdout:Total 18 MB/s | 99 MB 00:05 2024-02-27T07:30:34.227 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.1 MB 00:00 2024-02-27T07:30:34.459 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-27T07:30:34.845 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-27T07:30:34.845 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-27T07:30:36.058 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-02-27T07:30:36.224 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-27T07:30:36.224 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-27T07:30:37.476 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-02-27T07:30:39.148 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-27T07:30:39.808 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-27T07:30:39.942 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:30:42.610 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-02-27T07:30:42.723 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:30:42.904 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:30:44.851 INFO:teuthology.orchestra.run.smithi195.stderr:Modular dependency problems: 2024-02-27T07:30:44.851 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-02-27T07:30:44.851 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 1: conflicting requests 2024-02-27T07:30:44.851 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 2: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 3: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 4: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 5: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 6: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 7: conflicting requests 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.852 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 8: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 9: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 10: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 11: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 12: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 13: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 14: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 15: conflicting requests 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.853 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 16: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 17: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 18: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 19: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 20: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 21: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 22: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 23: conflicting requests 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.854 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 24: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 25: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 26: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 27: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 28: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 29: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 30: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 31: conflicting requests 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.855 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 32: conflicting requests 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 33: conflicting requests 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 34: conflicting requests 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 35: conflicting requests 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 36: conflicting requests 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 37: cannot install the best candidate for the job 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.856 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 38: conflicting requests 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 39: cannot install the best candidate for the job 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 40: cannot install the best candidate for the job 2024-02-27T07:30:44.857 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 41: cannot install the best candidate for the job 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 42: cannot install the best candidate for the job 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 43: cannot install the best candidate for the job 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.858 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: Problem 44: cannot install the best candidate for the job 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-27T07:30:44.859 INFO:teuthology.orchestra.run.smithi195.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-27T07:30:45.275 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-27T07:30:45.275 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-27T07:30:45.382 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-27T07:30:45.388 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-419.el9 baseos 6.0 M 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-419.el9 baseos 21 M 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-419.el9 baseos 39 M 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-419.el9 baseos 33 M 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:30:45.389 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-27T07:30:45.390 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-27T07:30:45.390 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-02-27T07:30:45.390 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:30:45.390 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 99 M 2024-02-27T07:30:45.390 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 122 M 2024-02-27T07:30:45.391 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-27T07:30:47.623 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:30:47.856 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:30:52.587 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:31:00.251 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 421 kB/s | 6.0 MB 00:14 2024-02-27T07:31:21.582 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 594 kB/s | 21 MB 00:35 2024-02-27T07:31:21.632 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:31:21.632 INFO:teuthology.orchestra.run.smithi149.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-27T07:31:21.632 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:31:21.719 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:31:22.210 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:31:22.210 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-27T07:31:22.210 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:31:22.211 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-27T07:31:22.658 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-27T07:31:22.659 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-27T07:31:22.659 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:31:22.659 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-27T07:31:23.146 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T07:31:23.289 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T07:31:23.289 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T07:31:23.290 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-27T07:31:23.290 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-27T07:31:23.290 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-27T07:31:23.290 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-27T07:31:23.290 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-02-27T07:31:23.290 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T07:31:23.412 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T07:31:23.412 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T07:31:23.413 DEBUG:teuthology.orchestra.run.smithi149:> sudo rpm -qi grub2-tools 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Name : grub2-tools 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Epoch : 1 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Version : 2.06 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Release : 68.el9 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: x86_64 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Group : Unspecified 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Size : 8264254 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:License : GPLv3+ 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-27T07:31:23.457 INFO:teuthology.orchestra.run.smithi149.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:Packager : builder@centos.org 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:Vendor : CentOS 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:Summary : Support tools for GRUB. 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:Description : 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:hardware devices. 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:31:23.458 INFO:teuthology.orchestra.run.smithi149.stdout:This subpackage provides tools for support of all platforms. 2024-02-27T07:31:23.459 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-27T07:31:23.459 INFO:teuthology.task.kernel:Updating grub on smithi149 to boot 5.14.0-419.el9.x86_64 2024-02-27T07:31:23.459 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-27T07:31:24.001 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-02-27T07:31:25.429 INFO:teuthology.orchestra.run.smithi149.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-02-27T07:31:25.468 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-02-27T07:31:25.484 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.UWHkT5HeX2 2024-02-27T07:31:25.484 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UWHkT5HeX2 2024-02-27T07:31:25.553 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.UWHkT5HeX2 2024-02-27T07:31:25.690 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.UWHkT5HeX2 is 6KB 2024-02-27T07:31:25.710 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.UWHkT5HeX2 2024-02-27T07:31:25.724 DEBUG:teuthology.orchestra.run.smithi149:> sudo /bin/ls /boot/loader/entries || true 2024-02-27T07:31:25.794 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-27T07:31:25.794 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-27T07:31:25.795 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-27T07:31:25.795 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-27T07:31:25.796 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-27T07:31:25.959 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-02-27T07:31:40.363 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 852 kB/s | 33 MB 00:40 2024-02-27T07:31:43.238 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 695 kB/s | 39 MB 00:57 2024-02-27T07:31:43.238 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-27T07:31:43.238 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.7 MB/s | 99 MB 00:57 2024-02-27T07:31:44.116 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-27T07:31:44.476 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-27T07:31:44.477 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-27T07:31:45.842 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-27T07:31:45.842 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-27T07:31:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-27T07:31:49.419 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-27T07:31:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:31:52.343 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:31:52.523 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:31:55.962 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-27T07:31:55.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:31:55.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:31:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:31:57.469 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:02.216 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:14.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-27T07:32:23.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:32:23.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:32:26.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-27T07:32:31.596 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:31.597 INFO:teuthology.orchestra.run.smithi195.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-27T07:32:31.597 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:32:31.683 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:32.153 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:32.153 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-27T07:32:32.153 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T07:32:32.153 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-27T07:32:32.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:32:32.601 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-27T07:32:33.126 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T07:32:33.270 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T07:32:33.270 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T07:32:33.270 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-27T07:32:33.270 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-27T07:32:33.270 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-27T07:32:33.270 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-27T07:32:33.271 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-02-27T07:32:33.271 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T07:32:33.392 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T07:32:33.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T07:32:33.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-27T07:32:33.430 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:32:33.431 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-02-27T07:32:33.432 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-27T07:32:33.432 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-419.el9.x86_64 2024-02-27T07:32:33.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-27T07:32:34.056 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-02-27T07:32:35.417 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-27T07:32:35.447 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-02-27T07:32:35.449 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-02-27T07:32:35.464 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.4aInzKLGwm 2024-02-27T07:32:35.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4aInzKLGwm 2024-02-27T07:32:35.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.4aInzKLGwm 2024-02-27T07:32:35.684 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.4aInzKLGwm is 6KB 2024-02-27T07:32:35.732 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.4aInzKLGwm 2024-02-27T07:32:35.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-02-27T07:32:35.820 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-27T07:32:35.820 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-27T07:32:35.820 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-27T07:32:35.820 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-27T07:32:35.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-27T07:32:35.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-02-27T07:32:38.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:32:38.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:32:57.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-27T07:33:05.976 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-27T07:33:05.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:33:05.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:33:12.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:33:12.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:33:12.411 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-02-27T07:33:12.888 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:33:12.888 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-27T07:33:12.889 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-02-27T07:33:12.903 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-419.el9.x86_64 2024-02-27T07:33:12.904 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-27T07:33:12.904 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-27T07:33:12.904 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-27T07:33:13.905 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-27T07:33:13.905 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-27T07:33:14.001 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-02-27T07:33:14.042 DEBUG:teuthology.parallel:result is None 2024-02-27T07:33:24.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-02-27T07:33:33.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:33:33.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:33:36.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-02-27T07:33:48.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:33:48.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:34:20.340 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-27T07:34:20.746 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:34:20.746 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-27T07:34:20.746 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-27T07:34:20.760 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-419.el9.x86_64 2024-02-27T07:34:20.760 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-27T07:34:20.760 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-27T07:34:20.761 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-27T07:34:21.761 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-27T07:34:21.761 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-27T07:34:21.924 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-02-27T07:34:21.966 DEBUG:teuthology.parallel:result is None 2024-02-27T07:34:21.966 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-27T07:34:21.974 INFO:teuthology.task.internal:Creating test directory... 2024-02-27T07:34:21.974 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-27T07:34:21.977 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-27T07:34:21.993 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-27T07:34:22.026 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-27T07:34:22.058 INFO:teuthology.task.internal:Creating archive directory... 2024-02-27T07:34:22.058 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-27T07:34:22.061 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-27T07:34:22.113 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-27T07:34:22.121 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-27T07:34:22.122 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-27T07:34:22.124 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-27T07:34:22.164 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T07:34:22.179 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T07:34:22.503 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T07:34:22.518 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T07:34:22.520 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-27T07:34:22.528 INFO:teuthology.task.internal:Configuring sudo... 2024-02-27T07:34:22.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-27T07:34:22.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-27T07:34:22.574 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-27T07:34:22.584 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-27T07:34:22.584 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-27T07:34:22.604 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-27T07:34:22.635 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T07:34:22.704 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T07:34:22.767 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T07:34:22.809 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T07:34:22.854 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:34:22.855 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-27T07:34:22.924 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T07:34:22.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T07:34:23.047 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T07:34:23.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T07:34:23.161 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:34:23.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-27T07:34:23.205 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-02-27T07:34:23.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-02-27T07:34:23.260 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T07:34:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T07:34:23.740 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-27T07:34:23.751 INFO:teuthology.task.internal:Starting timer... 2024-02-27T07:34:23.751 INFO:teuthology.run_tasks:Running task pcp... 2024-02-27T07:34:23.793 INFO:teuthology.run_tasks:Running task selinux... 2024-02-27T07:34:23.838 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-02-27T07:34:23.901 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-02-27T07:34:23.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-02-27T07:34:24.039 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-02-27T07:34:24.041 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-27T07:34:24.041 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-02-27T07:34:24.060 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-02-27T07:34:24.061 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-02-27T07:34:24.168 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-02-27T07:34:24.168 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi149.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-02-27T07:34:24.168 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T07:34:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T07:34:24.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T07:34:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T07:34:24.224 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-27T07:34:24.224 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-02-27T07:34:24.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-02-27T07:34:24.362 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-27T07:34:24.372 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-27T07:34:24.378 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-27T07:34:24.501 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-27T07:34:24.520 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-27T07:34:24.522 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi149.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-27T07:38:57.779 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-02-27T07:38:57.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:38:57.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:38:57.873 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-02-27T07:38:57.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-27T07:38:57.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:38:57.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T07:38:58.041 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-27T07:38:58.119 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-27T07:38:58.120 INFO:teuthology.run_tasks:Running task clock... 2024-02-27T07:38:58.132 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-27T07:38:58.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-27T07:38:58.132 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T07:38:58.134 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-27T07:38:58.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T07:38:58.163 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-27T07:38:58.179 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-27T07:38:58.191 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-27T07:38:58.208 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-27T07:38:58.209 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-02-27T07:38:58.222 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-02-27T07:38:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-02-27T07:38:58.238 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-27T07:38:58.251 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-02-27T07:38:58.255 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-27T07:38:58.268 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-27T07:38:58.284 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-27T07:38:58.305 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-02-27T07:38:58.308 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T07:38:58.308 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-02-27T07:38:58.308 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.309 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.309 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.309 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.345 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.349 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T07:38:58.350 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-27T07:38:58.449 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'} 2024-02-27T07:38:58.449 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-27T07:38:58.449 INFO:tasks.cephadm:Cluster fsid is 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:38:58.450 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-27T07:38:58.450 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.149', 'mon.c': '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'mon.b': '172.21.15.195'} 2024-02-27T07:38:58.450 INFO:tasks.cephadm:First mon is mon.a on smithi149 2024-02-27T07:38:58.450 INFO:tasks.cephadm:First mgr is y 2024-02-27T07:38:58.450 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-27T07:38:58.450 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2024-02-27T07:38:58.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-02-27T07:38:58.507 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-27T07:38:58.507 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:58.769 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 27 07:38 /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:58.769 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:59.035 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 27 07:38 /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:59.035 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:59.051 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-27T07:38:59.077 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-27T07:38:59.077 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-27T07:38:59.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-27T07:39:00.000 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T07:39:00.026 INFO:teuthology.orchestra.run.smithi195.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T07:39:17.317 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: "repo_digests": [ 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-02-27T07:39:17.318 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:39:19.465 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-27T07:39:19.465 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout: "repo_digests": [ 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-27T07:39:19.466 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-27T07:39:19.484 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2024-02-27T07:39:19.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-02-27T07:39:19.553 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2024-02-27T07:39:19.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-02-27T07:39:19.619 INFO:tasks.cephadm:Writing seed config... 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-27T07:39:19.621 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-27T07:39:19.622 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:39:19.622 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-27T07:39:19.647 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 448a3a16-d543-11ee-95c3-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-27T07:39:19.647 DEBUG:teuthology.orchestra.run.smithi149:mon.a> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service 2024-02-27T07:39:19.690 DEBUG:teuthology.orchestra.run.smithi149:mgr.y> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service 2024-02-27T07:39:19.732 INFO:tasks.cephadm:Bootstrapping... 2024-02-27T07:39:19.732 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 448a3a16-d543-11ee-95c3-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.149 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-27T07:39:19.935 INFO:teuthology.orchestra.run.smithi149.stderr:-------------------------------------------------------------------------------- 2024-02-27T07:39:19.935 INFO:teuthology.orchestra.run.smithi149.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '448a3a16-d543-11ee-95c3-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.149', '--skip-admin-label'] 2024-02-27T07:39:19.958 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: 4.9.3-dev 2024-02-27T07:39:19.961 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying podman|docker is present... 2024-02-27T07:39:19.984 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: 4.9.3-dev 2024-02-27T07:39:19.987 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying lvm2 is present... 2024-02-27T07:39:19.987 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying time synchronization is in place... 2024-02-27T07:39:19.994 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-27T07:39:20.001 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:39:20.008 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: enabled 2024-02-27T07:39:20.014 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: active 2024-02-27T07:39:20.015 INFO:teuthology.orchestra.run.smithi149.stderr:Unit chronyd.service is enabled and running 2024-02-27T07:39:20.015 INFO:teuthology.orchestra.run.smithi149.stderr:Repeating the final host check... 2024-02-27T07:39:20.042 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: 4.9.3-dev 2024-02-27T07:39:20.044 INFO:teuthology.orchestra.run.smithi149.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-27T07:39:20.044 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl is present 2024-02-27T07:39:20.044 INFO:teuthology.orchestra.run.smithi149.stderr:lvcreate is present 2024-02-27T07:39:20.050 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-27T07:39:20.057 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:39:20.064 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: enabled 2024-02-27T07:39:20.071 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: active 2024-02-27T07:39:20.072 INFO:teuthology.orchestra.run.smithi149.stderr:Unit chronyd.service is enabled and running 2024-02-27T07:39:20.072 INFO:teuthology.orchestra.run.smithi149.stderr:Host looks OK 2024-02-27T07:39:20.072 INFO:teuthology.orchestra.run.smithi149.stderr:Cluster fsid: 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:20.072 INFO:teuthology.orchestra.run.smithi149.stderr:Acquiring lock 139758903213552 on /run/cephadm/448a3a16-d543-11ee-95c3-87774f69a715.lock 2024-02-27T07:39:20.072 INFO:teuthology.orchestra.run.smithi149.stderr:Lock 139758903213552 acquired on /run/cephadm/448a3a16-d543-11ee-95c3-87774f69a715.lock 2024-02-27T07:39:20.073 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying IP 172.21.15.149 port 3300 ... 2024-02-27T07:39:20.073 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying IP 172.21.15.149 port 6789 ... 2024-02-27T07:39:20.073 INFO:teuthology.orchestra.run.smithi149.stderr:Base mon IP is 172.21.15.149, final addrv is [v2:172.21.15.149:3300,v1:172.21.15.149:6789] 2024-02-27T07:39:20.076 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.149 metric 100 2024-02-27T07:39:20.076 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.149 metric 100 2024-02-27T07:39:20.078 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-27T07:39:20.079 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-27T07:39:20.079 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1762sec hoplimit 64 pref medium 2024-02-27T07:39:20.081 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-27T07:39:20.081 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-27T07:39:20.081 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-27T07:39:20.081 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-27T07:39:20.081 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7d57/64 scope link 2024-02-27T07:39:20.082 INFO:teuthology.orchestra.run.smithi149.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-27T07:39:20.082 INFO:teuthology.orchestra.run.smithi149.stderr:Mon IP `172.21.15.149` is in CIDR network `172.21.0.0/20` 2024-02-27T07:39:20.082 INFO:teuthology.orchestra.run.smithi149.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-27T07:39:20.083 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T07:39:20.186 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Getting image source signatures 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-27T07:39:20.554 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-27T07:39:20.555 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-27T07:39:20.671 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: Writing manifest to image destination 2024-02-27T07:39:20.823 INFO:teuthology.orchestra.run.smithi149.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-27T07:39:21.193 INFO:teuthology.orchestra.run.smithi149.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-27T07:39:21.881 INFO:teuthology.orchestra.run.smithi149.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-27T07:39:21.881 INFO:teuthology.orchestra.run.smithi149.stderr:Extracting ceph user uid/gid from container image... 2024-02-27T07:39:22.247 INFO:teuthology.orchestra.run.smithi149.stderr:stat: 167 167 2024-02-27T07:39:22.903 INFO:teuthology.orchestra.run.smithi149.stderr:Creating initial keys... 2024-02-27T07:39:23.352 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph-authtool: AQCrkd1ls4yDDxAAI5L1lY3CbYkl+aQJJduxzA== 2024-02-27T07:39:24.174 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph-authtool: AQCskd1lovrDBBAA0xYDOJpP6SvAF/xPbbtVfw== 2024-02-27T07:39:25.130 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph-authtool: AQCtkd1lc58dAxAAIkAlM5uUJnszYpvPFmRPeg== 2024-02-27T07:39:25.553 INFO:teuthology.orchestra.run.smithi149.stderr:Creating initial monmap... 2024-02-27T07:39:25.952 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-27T07:39:25.952 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-27T07:39:25.952 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:25.953 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr:monmaptool for a [v2:172.21.15.149:3300,v1:172.21.15.149:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr:setting min_mon_release = octopus 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: set fsid to 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:39:26.434 INFO:teuthology.orchestra.run.smithi149.stderr:Creating mon... 2024-02-27T07:39:27.480 INFO:teuthology.orchestra.run.smithi149.stderr:create mon.a on 2024-02-27T07:39:27.675 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-27T07:39:27.823 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target. 2024-02-27T07:39:27.823 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target. 2024-02-27T07:39:28.121 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Failed to reset failed state of unit ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service: Unit ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service not loaded. 2024-02-27T07:39:28.128 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Created symlink /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target.wants/ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service → /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715@.service. 2024-02-27T07:39:28.779 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:28 smithi149 podman[24426]: 2024-02-27 07:39:28.701561178 +0000 UTC m=+0.295158943 container start 5071d6619c91b6a265433eecc410ab1bd17831209bf24e6b8f71230eff6dfa7a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, version=8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-02-27T07:39:28.779 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:28 smithi149 ceph-mon[24445]: mkfs 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:28.819 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: disabled 2024-02-27T07:39:28.825 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:39:28.826 INFO:teuthology.orchestra.run.smithi149.stderr:firewalld.service is not enabled 2024-02-27T07:39:28.826 INFO:teuthology.orchestra.run.smithi149.stderr:Not possible to enable service . firewalld.service is not available 2024-02-27T07:39:28.826 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mon to start... 2024-02-27T07:39:28.826 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mon... 2024-02-27T07:39:29.053 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:28 smithi149 ceph-mon[24445]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-27T07:39:29.053 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:28 smithi149 bash[24426]: 5071d6619c91b6a265433eecc410ab1bd17831209bf24e6b8f71230eff6dfa7a 2024-02-27T07:39:29.053 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:28 smithi149 systemd[1]: Started Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: cluster: 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: id: 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: services: 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.555749s) 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: data: 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-27T07:39:29.322 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-27T07:39:29.323 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-27T07:39:29.323 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: pgs: 2024-02-27T07:39:29.323 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:29.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-27T07:39:29.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: monmap e1: 1 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0]} 2024-02-27T07:39:29.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: fsmap 2024-02-27T07:39:29.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: osdmap e1: 0 total, 0 up, 0 in 2024-02-27T07:39:29.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: mgrmap e1: no daemons active 2024-02-27T07:39:29.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:29 smithi149 ceph-mon[24445]: from='client.? 172.21.15.149:0/1796788171' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-27T07:39:29.907 INFO:teuthology.orchestra.run.smithi149.stderr:mon is available 2024-02-27T07:39:29.907 INFO:teuthology.orchestra.run.smithi149.stderr:Assimilating anything we can from ceph.conf... 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [global] 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: fsid = 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.149:3300,v1:172.21.15.149:6789] 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [mgr] 2024-02-27T07:39:30.397 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-02-27T07:39:30.398 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-27T07:39:30.398 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:30.398 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [osd] 2024-02-27T07:39:30.398 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-27T07:39:30.398 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-27T07:39:31.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:30 smithi149 ceph-mon[24445]: from='client.? 172.21.15.149:0/821060665' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-27T07:39:31.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:30 smithi149 ceph-mon[24445]: from='client.? 172.21.15.149:0/821060665' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-27T07:39:34.833 INFO:teuthology.orchestra.run.smithi149.stderr:Generating new minimal ceph.conf... 2024-02-27T07:39:38.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:38 smithi149 ceph-mon[24445]: from='client.? 172.21.15.149:0/1413406335' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:39:41.329 INFO:teuthology.orchestra.run.smithi149.stderr:Restarting the monitor... 2024-02-27T07:39:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 systemd[1]: Stopping Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:39:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 bash[24676]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon.a" found: no such container 2024-02-27T07:39:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[24441]: 2024-02-27T07:39:41.468+0000 7f5f34dc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:39:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[24441]: 2024-02-27T07:39:41.468+0000 7f5f34dc7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-27T07:39:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 podman[24687]: 2024-02-27 07:39:41.540411183 +0000 UTC m=+0.152855782 container died 5071d6619c91b6a265433eecc410ab1bd17831209bf24e6b8f71230eff6dfa7a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, release=754) 2024-02-27T07:39:42.062 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 podman[24687]: 2024-02-27 07:39:41.809798921 +0000 UTC m=+0.422243514 container cleanup 5071d6619c91b6a265433eecc410ab1bd17831209bf24e6b8f71230eff6dfa7a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, version=8, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, release=754, distribution-scope=public, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-27T07:39:42.062 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 bash[24687]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a 2024-02-27T07:39:42.062 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:41 smithi149 bash[24713]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon.a" found: no such container 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 podman[24701]: 2024-02-27 07:39:42.060606866 +0000 UTC m=+0.518033586 container remove 5071d6619c91b6a265433eecc410ab1bd17831209bf24e6b8f71230eff6dfa7a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.buildah.version=1.19.8) 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 systemd[1]: Stopped Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 systemd[1]: Starting Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 podman[24777]: 2024-02-27 07:39:42.202960313 +0000 UTC m=+0.018630039 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-27T07:39:42.382 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 podman[24777]: 2024-02-27 07:39:42.30316859 +0000 UTC m=+0.118838299 container create 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-02-27T07:39:42.535 INFO:teuthology.orchestra.run.smithi149.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-27T07:39:42.660 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 podman[24777]: 2024-02-27 07:39:42.455680311 +0000 UTC m=+0.271350026 container init 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, maintainer=Guillaume Abrioux , vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8) 2024-02-27T07:39:42.660 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 podman[24777]: 2024-02-27 07:39:42.458618945 +0000 UTC m=+0.274288652 container start 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, name=centos-stream, distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: pidfile_write: ignore empty --pid-file 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: load: jerasure load: lrc 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: RocksDB version: 6.15.5 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Compile date Apr 18 2022 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: DB SUMMARY 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: DB Session ID: JZ9XICNDPZO0A18WL5LX 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: CURRENT file: CURRENT 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-27T07:39:42.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83657 ; 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.error_if_exists: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.create_if_missing: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.paranoid_checks: 1 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.env: 0x55a62fac6860 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.fs: Posix File System 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.info_log: 0x55a6303f1dc0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.statistics: (nil) 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.use_fsync: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_log_file_size: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T07:39:42.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_fallocate: 1 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.use_direct_reads: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.db_log_dir: 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T07:39:42.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.write_buffer_manager: 0x55a6304e2240 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T07:39:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.unordered_write: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.row_cache: None 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.wal_filter: None 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.preserve_deletes: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.two_write_queues: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.atomic_flush: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T07:39:42.665 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.log_readahead_size: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_background_jobs: 2 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_background_compactions: -1 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_subcompactions: 1 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T07:39:42.666 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T07:39:42.667 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_open_files: -1 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_background_flushes: -1 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Compression algorithms supported: 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T07:39:42.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kZSTD supported: 0 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kXpressCompression supported: 0 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kLZ4Compression supported: 1 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kBZip2Compression supported: 0 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kZlibCompression supported: 1 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: kSnappyCompression supported: 1 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.merge_operator: 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_filter: None 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T07:39:42.669 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a6303bdd00) 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_type: 0 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: data_block_index_type: 0 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_shortening: 1 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: hash_index_allow_collision: 1 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: checksum: 1 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: no_block_cache: 0 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache: 0x55a630428170 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_options: 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: capacity : 536870912 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: num_shard_bits : 4 2024-02-27T07:39:42.670 INFO:journalctl@ceph.mon.a.smithi149.stdout: strict_capacity_limit : 0 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_compressed: (nil) 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: persistent_cache: (nil) 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_size: 4096 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_size_deviation: 10 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_restart_interval: 16 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_block_restart_interval: 1 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: metadata_block_size: 4096 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: partition_filters: 0 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: use_delta_encoding: 1 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: whole_key_filtering: 1 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: verify_compression: 0 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: format_version: 4 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: enable_index_compression: 1 2024-02-27T07:39:42.671 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_align: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression: NoCompression 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.num_levels: 7 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:39:42.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T07:39:42.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.arena_block_size: 4194304 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-27T07:39:42.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.table_properties_collectors: 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.inplace_update_support: 0 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T07:39:42.675 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.bloom_locality: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.max_successive_merges: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.ttl: 2592000 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.enable_blob_files: false 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.min_blob_size: 0 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T07:39:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019582483899, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019582484674, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79755, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78001, "index_size": 242, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10059, "raw_average_key_size": 47, "raw_value_size": 72265, "raw_average_value_size": 339, "num_data_blocks": 11, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709019582, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "b07cc65d-8bef-41c0-9129-ce2727e30d09", "db_session_id": "JZ9XICNDPZO0A18WL5LX"}} 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019582484996, "job": 1, "event": "recovery_finished"} 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a63040ea80 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: DB pointer 0x55a63041e000 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-27T07:39:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** DB Stats ** 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: L0 2/0 79.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Sum 2/0 79.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative compaction: 0.00 GB write, 25.91 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval compaction: 0.00 GB write, 25.91 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:39:42.679 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: L0 2/0 79.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Sum 2/0 79.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.4 0.00 0.00 1 0.001 0 0 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T07:39:42.680 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative compaction: 0.00 GB write, 25.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: starting mon.a rank 0 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???) e1 preinit fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).mds e1 new map 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).mds e1 print_map 2024-02-27T07:39:42.681 INFO:journalctl@ceph.mon.a.smithi149.stdout: e1 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout: legacy client fscid: -1 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout: No filesystems configured 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expand map: {default=false} 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta from 'false' to 'false' 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expanded map: {default=false} 2024-02-27T07:39:42.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expand map: {default=info} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta from 'info' to 'info' 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expanded map: {default=info} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expand map: {default=daemon} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta from 'daemon' to 'daemon' 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expanded map: {default=daemon} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expand map: {default=debug} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta from 'debug' to 'debug' 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: expand_channel_meta expanded map: {default=debug} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 bash[24777]: 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 systemd[1]: Started Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: monmap e1: 1 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0]} 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: fsmap 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: osdmap e1: 0 total, 0 up, 0 in 2024-02-27T07:39:42.683 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:42 smithi149 ceph-mon[24796]: mgrmap e1: no daemons active 2024-02-27T07:39:43.591 INFO:teuthology.orchestra.run.smithi149.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-27T07:39:43.592 INFO:teuthology.orchestra.run.smithi149.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-27T07:39:43.592 INFO:teuthology.orchestra.run.smithi149.stderr:Creating mgr... 2024-02-27T07:39:43.592 INFO:teuthology.orchestra.run.smithi149.stderr:Verifying port 9283 ... 2024-02-27T07:39:43.765 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Failed to reset failed state of unit ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service: Unit ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service not loaded. 2024-02-27T07:39:43.771 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: Created symlink /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715.target.wants/ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-448a3a16-d543-11ee-95c3-87774f69a715@.service. 2024-02-27T07:39:44.262 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:44 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1225519084' entity='client.admin' 2024-02-27T07:39:44.455 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: disabled 2024-02-27T07:39:44.461 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:39:44.462 INFO:teuthology.orchestra.run.smithi149.stderr:firewalld.service is not enabled 2024-02-27T07:39:44.462 INFO:teuthology.orchestra.run.smithi149.stderr:Not possible to enable service . firewalld.service is not available 2024-02-27T07:39:44.470 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: disabled 2024-02-27T07:39:44.477 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:39:44.478 INFO:teuthology.orchestra.run.smithi149.stderr:firewalld.service is not enabled 2024-02-27T07:39:44.478 INFO:teuthology.orchestra.run.smithi149.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-27T07:39:44.478 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mgr to start... 2024-02-27T07:39:44.478 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mgr... 2024-02-27T07:39:44.957 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:44.954+0000 7f336318e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:39:45.000 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsid": "448a3a16-d543-11ee-95c3-87774f69a715", 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "health": { 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 0 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:45.001 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "a" 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T07:39:45.002 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T07:39:45.006 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "available": false, 2024-02-27T07:39:45.007 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "iostat", 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "nfs", 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "restful" 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modified": "2024-02-27T07:39:28.766427+0000", 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:45.008 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T07:39:45.009 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:39:45.315 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1799279352' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T07:39:45.601 INFO:teuthology.orchestra.run.smithi149.stderr:mgr not available, waiting (1/15)... 2024-02-27T07:39:45.889 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:45.569+0000 7f336318e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:39:45.889 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:45.659+0000 7f336318e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:39:45.889 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:45.739+0000 7f336318e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:39:46.155 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:45.886+0000 7f336318e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:39:46.155 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:45.965+0000 7f336318e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:39:46.155 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:46.071+0000 7f336318e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:39:46.155 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:46.152+0000 7f336318e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:39:46.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:46.315+0000 7f336318e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:39:46.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:46.398+0000 7f336318e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:39:47.731 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:47.432+0000 7f336318e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:39:47.732 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:47.558+0000 7f336318e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:39:47.985 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:47.753+0000 7f336318e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:39:47.985 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:47.839+0000 7f336318e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:39:47.985 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:47.975+0000 7f336318e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:39:48.164 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:48.164 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:39:48.164 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsid": "448a3a16-d543-11ee-95c3-87774f69a715", 2024-02-27T07:39:48.164 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "health": { 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 0 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "a" 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T07:39:48.165 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T07:39:48.166 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T07:39:48.169 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T07:39:48.169 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T07:39:48.169 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.169 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T07:39:48.170 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "available": false, 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "iostat", 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "nfs", 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "restful" 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modified": "2024-02-27T07:39:28.766427+0000", 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:48.171 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:48.172 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T07:39:48.172 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:39:48.615 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:48 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/619124099' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T07:39:48.615 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:48 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:48.310+0000 7f336318e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:39:48.615 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:48 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:48.530+0000 7f336318e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:39:48.616 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:48 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:48.613+0000 7f336318e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:39:48.745 INFO:teuthology.orchestra.run.smithi149.stderr:mgr not available, waiting (2/15)... 2024-02-27T07:39:48.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:48 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:48.788+0000 7f336318e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:39:49.987 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:49 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:49.698+0000 7f336318e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:39:49.987 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:49 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:49.984+0000 7f336318e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:39:50.289 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:50.077+0000 7f336318e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:39:50.289 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:50.223+0000 7f336318e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:39:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: Activating manager daemon y 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: mgrmap e2: y(active, starting, since 0.00286922s) 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: Manager daemon y is now available 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' 2024-02-27T07:39:50.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:50 smithi149 ceph-mon[24796]: from='mgr.14100 172.21.15.149:0/2738632487' entity='mgr.y' 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsid": "448a3a16-d543-11ee-95c3-87774f69a715", 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "health": { 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T07:39:51.265 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 0 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "a" 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T07:39:51.266 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T07:39:51.267 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T07:39:51.270 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T07:39:51.271 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "available": true, 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "iostat", 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "nfs", 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "restful" 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ], 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "modified": "2024-02-27T07:39:28.766427+0000", 2024-02-27T07:39:51.272 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "services": {} 2024-02-27T07:39:51.273 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: }, 2024-02-27T07:39:51.273 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T07:39:51.273 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:39:51.826 INFO:teuthology.orchestra.run.smithi149.stderr:mgr is available 2024-02-27T07:39:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:52 smithi149 ceph-mon[24796]: mgrmap e3: y(active, since 1.00612s) 2024-02-27T07:39:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:52 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4159439334' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T07:39:52.444 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:52.445 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [global] 2024-02-27T07:39:52.445 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: fsid = 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [mgr] 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: [osd] 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-27T07:39:52.446 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-27T07:39:53.025 INFO:teuthology.orchestra.run.smithi149.stderr:Enabling cephadm module... 2024-02-27T07:39:53.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/504796459' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-27T07:39:53.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/504796459' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-27T07:39:54.552 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-mon[24796]: mgrmap e4: y(active, since 3s) 2024-02-27T07:39:54.553 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/180588812' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-27T07:39:54.553 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setuser ceph since I am not root 2024-02-27T07:39:54.553 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setgroup ceph since I am not root 2024-02-27T07:39:54.829 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:54.737+0000 7f4a5f0d6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:39:55.164 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:54.827+0000 7f4a5f0d6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:39:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/180588812' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-27T07:39:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-mon[24796]: mgrmap e5: y(active, since 4s) 2024-02-27T07:39:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:39:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-27T07:39:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "available": true, 2024-02-27T07:39:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-27T07:39:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-27T07:39:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:39:55.785 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.444+0000 7f4a5f0d6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:39:55.785 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.547+0000 7f4a5f0d6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:39:55.785 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.631+0000 7f4a5f0d6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:39:56.063 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.786+0000 7f4a5f0d6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:39:56.063 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.867+0000 7f4a5f0d6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:39:56.063 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:55.977+0000 7f4a5f0d6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:39:56.063 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:56 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:56.060+0000 7f4a5f0d6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:39:56.088 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for the mgr to restart... 2024-02-27T07:39:56.088 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mgr epoch 5... 2024-02-27T07:39:56.328 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:39:56 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1380667574' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-27T07:39:56.328 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:56 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:56.237+0000 7f4a5f0d6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:39:56.328 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:56 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:56.326+0000 7f4a5f0d6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:39:57.678 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:57.363+0000 7f4a5f0d6000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:39:57.678 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:57.495+0000 7f4a5f0d6000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:39:58.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:57.684+0000 7f4a5f0d6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:39:58.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:57.770+0000 7f4a5f0d6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:39:58.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:57.905+0000 7f4a5f0d6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:39:58.534 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:58 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:58.225+0000 7f4a5f0d6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:39:58.534 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:58 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:58.447+0000 7f4a5f0d6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:39:58.534 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:58 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:58.531+0000 7f4a5f0d6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:39:58.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:58 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:58.699+0000 7f4a5f0d6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:39:59.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:59 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:59.640+0000 7f4a5f0d6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:40:00.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:39:59 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:39:59.931+0000 7f4a5f0d6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:40:00.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:00 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:00.025+0000 7f4a5f0d6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:40:00.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:00 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:00.174+0000 7f4a5f0d6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:40:01.252 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:40:01.252 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-27T07:40:01.252 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "initialized": true 2024-02-27T07:40:01.252 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:01] ENGINE Bus STARTING 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:01] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:01] ENGINE Bus STARTED 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: Active manager daemon y restarted 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: Activating manager daemon y 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: osdmap e2: 0 total, 0 up, 0 in 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: mgrmap e6: y(active, starting, since 0.0526186s) 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:40:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: Manager daemon y is now available 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:40:01.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:01 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:40:01.777 INFO:teuthology.orchestra.run.smithi149.stderr:mgr epoch 5 is available 2024-02-27T07:40:01.777 INFO:teuthology.orchestra.run.smithi149.stderr:Setting orchestrator backend to cephadm... 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:01] ENGINE Bus STARTING 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:01] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:01] ENGINE Bus STARTED 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: mgrmap e7: y(active, since 1.05532s) 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-27T07:40:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:02 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:03.400 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:03 smithi149 ceph-mon[24796]: mgrmap e8: y(active, since 2s) 2024-02-27T07:40:03.400 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:03 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:03.400 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:03 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:03.687 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: value unchanged 2024-02-27T07:40:04.289 INFO:teuthology.orchestra.run.smithi149.stderr:Generating ssh key... 2024-02-27T07:40:04.511 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:04 smithi149 ceph-mon[24796]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-mon[24796]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: Generating public/private rsa key pair. 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: Your identification has been saved in /tmp/tmpum3bsqxy/key. 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: Your public key has been saved in /tmp/tmpum3bsqxy/key.pub. 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: The key fingerprint is: 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: SHA256:3oFLybn62PXGyYtGV5KG5OxHV7O1NfoCMLh8KtaKc1o ceph-448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: The key's randomart image is: 2024-02-27T07:40:05.398 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: +---[RSA 3072]----+ 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | . | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | . o. o+| 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | . .+o. ...B| 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | + =+.=.oo | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | . S..o.+. | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | o = +o.o. . | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | oEo +.o= .. | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | o.o + ..o= | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: | .+ o.o...o. | 2024-02-27T07:40:05.399 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: +----[SHA256]-----+ 2024-02-27T07:40:06.233 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQgsFoCeQknVYh434JcKppMK4ECWYOenWcSB7sPCgofjzrQ2IxXNR5v84vNG2PIk5zKJTC4nuh9teuFyC9y6CdeqkNB/gza0t63a51gTzeFXXpZrJ5/S9sBMYFuAlTwrXMCwSNwjrQls4oTK7DiN4qkxm+RgkzeZJGNvw48RbQEqDOBYcTohE5KPMneHC92WucAAR+YmK8fUGQRzRFAIoFA722LrU3RDIZ+f6bR7XN2ALnNyMD90gmQqO48DJyUZDUW7GgNuvftKhUcNyRHjsQSkGatG+kiVHDaUzJ0+UFJIw6SKd3zTPnhwKo2zPvGHGeRNyQbRfuidiGG0nK4Co3pYBllqlmKtD174eiXdAUlKzj7r/pjlsHmBXXOb6hx7zKxK8NL//0oOomIlmk4lkK3WjEOx6zwGqQE/PlVrv2zlikbHN9Y4xk7c7JgrD7RzrXkll9FD7Recx8g2IJmbzyn1dl7D0RcjHqpcj9UkrrLdnmMrjDbpOF9vi3/bCD/50= ceph-448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:06 smithi149 ceph-mon[24796]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:06.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:06 smithi149 ceph-mon[24796]: Generating ssh key... 2024-02-27T07:40:06.869 INFO:teuthology.orchestra.run.smithi149.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-27T07:40:06.869 INFO:teuthology.orchestra.run.smithi149.stderr:Adding key to root@localhost authorized_keys... 2024-02-27T07:40:06.869 INFO:teuthology.orchestra.run.smithi149.stderr:Adding host smithi149... 2024-02-27T07:40:07.410 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:07 smithi149 ceph-mon[24796]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:08.604 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: Added host 'smithi149' with addr '172.21.15.149' 2024-02-27T07:40:08.615 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:08 smithi149 ceph-mon[24796]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "addr": "172.21.15.149", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:08.616 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:08 smithi149 ceph-mon[24796]: Deploying cephadm binary to smithi149 2024-02-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi149.stderr:Deploying unmanaged mon service... 2024-02-27T07:40:09.825 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:09 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:09.825 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:09 smithi149 ceph-mon[24796]: Added host smithi149 2024-02-27T07:40:09.825 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:09 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:09.825 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-27T07:40:10.391 INFO:teuthology.orchestra.run.smithi149.stderr:Deploying unmanaged mgr service... 2024-02-27T07:40:11.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:10 smithi149 ceph-mon[24796]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:11.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:10 smithi149 ceph-mon[24796]: Saving service mon spec with placement count:5 2024-02-27T07:40:11.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:10 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:11.210 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-27T07:40:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:12 smithi149 ceph-mon[24796]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:12.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:12 smithi149 ceph-mon[24796]: Saving service mgr spec with placement count:2 2024-02-27T07:40:12.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:12 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:12.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:12 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:13.886 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:13 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:13.887 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:13 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/31153829' entity='client.admin' 2024-02-27T07:40:15.213 INFO:teuthology.orchestra.run.smithi149.stderr:Enabling the dashboard module... 2024-02-27T07:40:15.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:15 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/561931230' entity='client.admin' 2024-02-27T07:40:16.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setuser ceph since I am not root 2024-02-27T07:40:16.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setgroup ceph since I am not root 2024-02-27T07:40:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:16 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:16 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1184686044' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-27T07:40:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:16 smithi149 ceph-mon[24796]: from='mgr.14118 172.21.15.149:0/1917389943' entity='mgr.y' 2024-02-27T07:40:17.277 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:17.186+0000 7f9919826000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:40:17.277 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:17.274+0000 7f9919826000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:40:17.901 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1184686044' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-27T07:40:17.901 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-mon[24796]: mgrmap e9: y(active, since 16s) 2024-02-27T07:40:17.989 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:40:17.990 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-27T07:40:17.990 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "available": true, 2024-02-27T07:40:17.990 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-27T07:40:17.990 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-27T07:40:17.990 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:40:18.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:17.898+0000 7f9919826000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:40:18.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:17.991+0000 7f9919826000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:40:18.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.081+0000 7f9919826000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:40:18.509 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.231+0000 7f9919826000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:40:18.509 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.310+0000 7f9919826000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:40:18.509 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.422+0000 7f9919826000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:40:18.509 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.506+0000 7f9919826000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:40:18.760 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3582710907' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-27T07:40:18.761 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.671+0000 7f9919826000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:40:18.761 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:18.758+0000 7f9919826000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:40:19.016 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for the mgr to restart... 2024-02-27T07:40:19.017 INFO:teuthology.orchestra.run.smithi149.stderr:Waiting for mgr epoch 9... 2024-02-27T07:40:19.834 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:19.809+0000 7f9919826000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:40:20.135 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:19.941+0000 7f9919826000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:40:20.135 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:20.132+0000 7f9919826000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:40:20.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:20.220+0000 7f9919826000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:40:20.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:20.359+0000 7f9919826000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:40:20.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:20.705+0000 7f9919826000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:40:21.202 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:20.939+0000 7f9919826000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:40:21.202 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:21 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:21.026+0000 7f9919826000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:40:21.202 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:21 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:21.200+0000 7f9919826000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:40:22.425 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:22.130+0000 7f9919826000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:40:22.678 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:22.423+0000 7f9919826000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:40:22.678 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:22.519+0000 7f9919826000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:40:22.678 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:40:22.667+0000 7f9919826000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: Active manager daemon y restarted 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: Activating manager daemon y 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: osdmap e3: 0 total, 0 up, 0 in 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: mgrmap e10: y(active, starting, since 0.816693s) 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: Manager daemon y is now available 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:40:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:40:24.513 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: { 2024-02-27T07:40:24.513 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-27T07:40:24.513 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: "initialized": true 2024-02-27T07:40:24.514 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: } 2024-02-27T07:40:24.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:24] ENGINE Bus STARTING 2024-02-27T07:40:24.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:24] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:40:24.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:40:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:40:24] ENGINE Bus STARTED 2024-02-27T07:40:25.335 INFO:teuthology.orchestra.run.smithi149.stderr:mgr epoch 9 is available 2024-02-27T07:40:25.336 INFO:teuthology.orchestra.run.smithi149.stderr:Generating a dashboard self-signed certificate... 2024-02-27T07:40:25.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:25 smithi149 ceph-mon[24796]: mgrmap e11: y(active, since 1.81898s) 2024-02-27T07:40:25.595 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:25 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:24] ENGINE Bus STARTING 2024-02-27T07:40:25.595 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:26.055 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-27T07:40:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:24] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:40:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: [27/Feb/2024:07:40:24] ENGINE Bus STARTED 2024-02-27T07:40:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: mgrmap e12: y(active, since 2s) 2024-02-27T07:40:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:26.742 INFO:teuthology.orchestra.run.smithi149.stderr:Creating initial admin user... 2024-02-27T07:40:27.595 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:27 smithi149 ceph-mon[24796]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:27.595 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Nrb36cFOYHQeMvHZ7yVsmel8ano26n0vZh4mXeYEmkre2xgv1umH6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709019627, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-27T07:40:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:Fetching dashboard port number... 2024-02-27T07:40:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:28 smithi149 ceph-mon[24796]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:28 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: 8443 2024-02-27T07:40:29.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:29 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2479595060' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-27T07:40:29.973 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: disabled 2024-02-27T07:40:29.980 INFO:teuthology.orchestra.run.smithi149.stderr:systemctl: inactive 2024-02-27T07:40:29.981 INFO:teuthology.orchestra.run.smithi149.stderr:firewalld.service is not enabled 2024-02-27T07:40:29.981 INFO:teuthology.orchestra.run.smithi149.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr:Ceph Dashboard is now available at: 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr: URL: https://smithi149.front.sepia.ceph.com:8443/ 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr: User: admin 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr: Password: dtgzlqmyqs 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:29.983 INFO:teuthology.orchestra.run.smithi149.stderr:Enabling autotune for osd_memory_target 2024-02-27T07:40:30.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:30 smithi149 ceph-mon[24796]: mgrmap e13: y(active, since 6s) 2024-02-27T07:40:31.898 INFO:teuthology.orchestra.run.smithi149.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr:You can access the Ceph CLI with: 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-27T07:40:32.512 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr: ceph telemetry on 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr:For more information see: 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-27T07:40:32.513 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T07:40:32.514 INFO:teuthology.orchestra.run.smithi149.stderr:Bootstrap complete. 2024-02-27T07:40:32.548 INFO:tasks.cephadm:Fetching config... 2024-02-27T07:40:32.549 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:40:32.549 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-27T07:40:32.565 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-27T07:40:32.565 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:40:32.565 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-27T07:40:32.620 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-27T07:40:32.620 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:40:32.620 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-27T07:40:32.689 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-27T07:40:32.689 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:40:32.689 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-27T07:40:32.744 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-27T07:40:32.744 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQgsFoCeQknVYh434JcKppMK4ECWYOenWcSB7sPCgofjzrQ2IxXNR5v84vNG2PIk5zKJTC4nuh9teuFyC9y6CdeqkNB/gza0t63a51gTzeFXXpZrJ5/S9sBMYFuAlTwrXMCwSNwjrQls4oTK7DiN4qkxm+RgkzeZJGNvw48RbQEqDOBYcTohE5KPMneHC92WucAAR+YmK8fUGQRzRFAIoFA722LrU3RDIZ+f6bR7XN2ALnNyMD90gmQqO48DJyUZDUW7GgNuvftKhUcNyRHjsQSkGatG+kiVHDaUzJ0+UFJIw6SKd3zTPnhwKo2zPvGHGeRNyQbRfuidiGG0nK4Co3pYBllqlmKtD174eiXdAUlKzj7r/pjlsHmBXXOb6hx7zKxK8NL//0oOomIlmk4lkK3WjEOx6zwGqQE/PlVrv2zlikbHN9Y4xk7c7JgrD7RzrXkll9FD7Recx8g2IJmbzyn1dl7D0RcjHqpcj9UkrrLdnmMrjDbpOF9vi3/bCD/50= ceph-448a3a16-d543-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-27T07:40:32.835 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQgsFoCeQknVYh434JcKppMK4ECWYOenWcSB7sPCgofjzrQ2IxXNR5v84vNG2PIk5zKJTC4nuh9teuFyC9y6CdeqkNB/gza0t63a51gTzeFXXpZrJ5/S9sBMYFuAlTwrXMCwSNwjrQls4oTK7DiN4qkxm+RgkzeZJGNvw48RbQEqDOBYcTohE5KPMneHC92WucAAR+YmK8fUGQRzRFAIoFA722LrU3RDIZ+f6bR7XN2ALnNyMD90gmQqO48DJyUZDUW7GgNuvftKhUcNyRHjsQSkGatG+kiVHDaUzJ0+UFJIw6SKd3zTPnhwKo2zPvGHGeRNyQbRfuidiGG0nK4Co3pYBllqlmKtD174eiXdAUlKzj7r/pjlsHmBXXOb6hx7zKxK8NL//0oOomIlmk4lkK3WjEOx6zwGqQE/PlVrv2zlikbHN9Y4xk7c7JgrD7RzrXkll9FD7Recx8g2IJmbzyn1dl7D0RcjHqpcj9UkrrLdnmMrjDbpOF9vi3/bCD/50= ceph-448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:32.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQgsFoCeQknVYh434JcKppMK4ECWYOenWcSB7sPCgofjzrQ2IxXNR5v84vNG2PIk5zKJTC4nuh9teuFyC9y6CdeqkNB/gza0t63a51gTzeFXXpZrJ5/S9sBMYFuAlTwrXMCwSNwjrQls4oTK7DiN4qkxm+RgkzeZJGNvw48RbQEqDOBYcTohE5KPMneHC92WucAAR+YmK8fUGQRzRFAIoFA722LrU3RDIZ+f6bR7XN2ALnNyMD90gmQqO48DJyUZDUW7GgNuvftKhUcNyRHjsQSkGatG+kiVHDaUzJ0+UFJIw6SKd3zTPnhwKo2zPvGHGeRNyQbRfuidiGG0nK4Co3pYBllqlmKtD174eiXdAUlKzj7r/pjlsHmBXXOb6hx7zKxK8NL//0oOomIlmk4lkK3WjEOx6zwGqQE/PlVrv2zlikbHN9Y4xk7c7JgrD7RzrXkll9FD7Recx8g2IJmbzyn1dl7D0RcjHqpcj9UkrrLdnmMrjDbpOF9vi3/bCD/50= ceph-448a3a16-d543-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-27T07:40:32.897 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQgsFoCeQknVYh434JcKppMK4ECWYOenWcSB7sPCgofjzrQ2IxXNR5v84vNG2PIk5zKJTC4nuh9teuFyC9y6CdeqkNB/gza0t63a51gTzeFXXpZrJ5/S9sBMYFuAlTwrXMCwSNwjrQls4oTK7DiN4qkxm+RgkzeZJGNvw48RbQEqDOBYcTohE5KPMneHC92WucAAR+YmK8fUGQRzRFAIoFA722LrU3RDIZ+f6bR7XN2ALnNyMD90gmQqO48DJyUZDUW7GgNuvftKhUcNyRHjsQSkGatG+kiVHDaUzJ0+UFJIw6SKd3zTPnhwKo2zPvGHGeRNyQbRfuidiGG0nK4Co3pYBllqlmKtD174eiXdAUlKzj7r/pjlsHmBXXOb6hx7zKxK8NL//0oOomIlmk4lkK3WjEOx6zwGqQE/PlVrv2zlikbHN9Y4xk7c7JgrD7RzrXkll9FD7Recx8g2IJmbzyn1dl7D0RcjHqpcj9UkrrLdnmMrjDbpOF9vi3/bCD/50= ceph-448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:32.912 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-27T07:40:33.071 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:32 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4031904166' entity='client.admin' 2024-02-27T07:40:36.581 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-27T07:40:36.581 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-27T07:40:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:36 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3247744349' entity='client.admin' 2024-02-27T07:40:38.251 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:38.251 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:40:38.251 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:38.251 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:38.251 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:39.406 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-02-27T07:40:39.406 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:40:39.406 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-02-27T07:40:39.425 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:40:39.425 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:40:39.482 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-02-27T07:40:39.482 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch host add smithi195 2024-02-27T07:40:40.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:40.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:40.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:40.047 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:40.047 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:40:40.047 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:40.047 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:39 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:40.783 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:40 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:40:40.783 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:40 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:40:42.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:41 smithi149 ceph-mon[24796]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:42.374 INFO:teuthology.orchestra.run.smithi149.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-02-27T07:40:43.011 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch host ls --format=json 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: Deploying cephadm binary to smithi195 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: Added host smithi195 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:43.625 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:44.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:44 smithi149 ceph-mon[24796]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:44.668 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:44 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:44.899 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:40:44.899 INFO:teuthology.orchestra.run.smithi149.stdout:[{"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-02-27T07:40:45.418 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-27T07:40:45.418 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd crush tunables default 2024-02-27T07:40:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:46 smithi149 ceph-mon[24796]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:40:46.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:46 smithi149 ceph-mon[24796]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:47.668 INFO:teuthology.orchestra.run.smithi149.stderr:adjusted tunables profile to default 2024-02-27T07:40:47.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:47.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:47.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:40:47.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:47.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:47 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2979277652' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-27T07:40:48.245 INFO:tasks.cephadm:Adding mon.a on smithi149 2024-02-27T07:40:48.246 INFO:tasks.cephadm:Adding mon.c on smithi149 2024-02-27T07:40:48.246 INFO:tasks.cephadm:Adding mon.b on smithi195 2024-02-27T07:40:48.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply mon '3;smithi149:172.21.15.149=a;smithi149:[v2:172.21.15.149:3301,v1:172.21.15.149:6790]=c;smithi195:172.21.15.195=b' 2024-02-27T07:40:48.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:48 smithi149 ceph-mon[24796]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:48.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:48 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2979277652' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-27T07:40:48.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:48 smithi149 ceph-mon[24796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T07:40:50.052 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-02-27T07:40:50.721 DEBUG:teuthology.orchestra.run.smithi149:mon.c> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.c.service 2024-02-27T07:40:50.723 DEBUG:teuthology.orchestra.run.smithi195:mon.b> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.b.service 2024-02-27T07:40:50.726 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-27T07:40:50.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T07:40:50.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:40:50.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:51.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:51 smithi149 ceph-mon[24796]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi149:172.21.15.149=a;smithi149:[v2:172.21.15.149:3301,v1:172.21.15.149:6790]=c;smithi195:172.21.15.195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:40:51.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:51 smithi149 ceph-mon[24796]: Saving service mon spec with placement smithi149:172.21.15.149=a;smithi149:[v2:172.21.15.149:3301,v1:172.21.15.149:6790]=c;smithi195:172.21.15.195=b;count:3 2024-02-27T07:40:51.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:51 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:40:51.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:51 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:40:51.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:51 smithi149 ceph-mon[24796]: Deploying daemon mon.c on smithi149 2024-02-27T07:40:52.430 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:40:52.430 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","modified":"2024-02-27T07:39:25.904582Z","created":"2024-02-27T07:39:25.904582Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T07:40:52.432 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-27T07:40:52.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:52 smithi149 ceph-mon[24796]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:52.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:52 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/1597767400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:40:53.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 systemd[1]: Starting Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:40:53.612 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 podman[29509]: 2024-02-27 07:40:53.410591415 +0000 UTC m=+0.018470699 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-27T07:40:53.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 podman[29509]: 2024-02-27 07:40:53.511731987 +0000 UTC m=+0.119611301 container create 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.component=centos-stream-container, release=754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-02-27T07:40:54.026 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-27T07:40:54.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 podman[29509]: 2024-02-27 07:40:53.75592644 +0000 UTC m=+0.363805709 container init 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 podman[29509]: 2024-02-27 07:40:53.759056179 +0000 UTC m=+0.366935449 container start 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, distribution-scope=public, RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754) 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: pidfile_write: ignore empty --pid-file 2024-02-27T07:40:54.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: load: jerasure load: lrc 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: RocksDB version: 6.15.5 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Compile date Apr 18 2022 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: DB SUMMARY 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: DB Session ID: G68UYVJFBPUL03HHNEHE 2024-02-27T07:40:54.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: CURRENT file: CURRENT 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.error_if_exists: 0 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.create_if_missing: 0 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.paranoid_checks: 1 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.env: 0x55bc7bac6860 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.fs: Posix File System 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.info_log: 0x55bc7d7dbdc0 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.statistics: (nil) 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.use_fsync: 0 2024-02-27T07:40:54.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_log_file_size: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_fallocate: 1 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.use_direct_reads: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.db_log_dir: 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T07:40:54.182 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.write_buffer_manager: 0x55bc7d8cc270 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T07:40:54.183 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.unordered_write: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.row_cache: None 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.wal_filter: None 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.preserve_deletes: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.two_write_queues: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.atomic_flush: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T07:40:54.184 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.log_readahead_size: 0 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_background_jobs: 2 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_background_compactions: -1 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_subcompactions: 1 2024-02-27T07:40:54.185 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_open_files: -1 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T07:40:54.186 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_background_flushes: -1 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Compression algorithms supported: 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kZSTD supported: 0 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kXpressCompression supported: 0 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kLZ4Compression supported: 1 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kBZip2Compression supported: 0 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kZlibCompression supported: 1 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: kSnappyCompression supported: 1 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T07:40:54.187 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.merge_operator: 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_filter: None 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc7d7a7d00) 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-02-27T07:40:54.188 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_type: 0 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: data_block_index_type: 0 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_shortening: 1 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: hash_index_allow_collision: 1 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: checksum: 1 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: no_block_cache: 0 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache: 0x55bc7d812170 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_options: 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: capacity : 536870912 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: num_shard_bits : 4 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: strict_capacity_limit : 0 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_compressed: (nil) 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: persistent_cache: (nil) 2024-02-27T07:40:54.189 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_size: 4096 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_size_deviation: 10 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_restart_interval: 16 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_block_restart_interval: 1 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: metadata_block_size: 4096 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: partition_filters: 0 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: use_delta_encoding: 1 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: whole_key_filtering: 1 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: verify_compression: 0 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: format_version: 4 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: enable_index_compression: 1 2024-02-27T07:40:54.190 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_align: 0 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression: NoCompression 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.num_levels: 7 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T07:40:54.191 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T07:40:54.192 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.arena_block_size: 4194304 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T07:40:54.193 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.table_properties_collectors: 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.inplace_update_support: 0 2024-02-27T07:40:54.194 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.bloom_locality: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.max_successive_merges: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.ttl: 2592000 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.enable_blob_files: false 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.min_blob_size: 0 2024-02-27T07:40:54.195 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019653784979, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019653785296, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709019653, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "050f5c90-7158-474c-a6be-0b79ce3fccfe", "db_session_id": "G68UYVJFBPUL03HHNEHE"}} 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709019653785596, "job": 1, "event": "recovery_finished"} 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T07:40:54.196 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55bc7d7f8a80 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: DB pointer 0x55bc7d808000 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** DB Stats ** 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:40:54.197 INFO:journalctl@ceph.mon.c.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:40:54.198 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T07:40:54.199 INFO:journalctl@ceph.mon.c.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-27T07:40:54.200 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: using public_addrv [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: starting mon.c rank -1 at public addrs [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] at bind addrs [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(???) e0 preinit fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).mds e1 new map 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: e1 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: legacy client fscid: -1 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout: No filesystems configured 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-27T07:40:54.201 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/1597767400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 ceph-mon[29527]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 bash[29509]: 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 2024-02-27T07:40:54.202 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:53 smithi149 systemd[1]: Started Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: mon.a calling monitor election 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: mon.c calling monitor election 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: monmap e2: 2 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: fsmap 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: mgrmap e13: y(active, since 36s) 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: overall HEALTH_OK 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:59.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: mon.a calling monitor election 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: mon.c calling monitor election 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: monmap e2: 2 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: fsmap 2024-02-27T07:40:59.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T07:40:59.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: mgrmap e13: y(active, since 36s) 2024-02-27T07:40:59.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: overall HEALTH_OK 2024-02-27T07:40:59.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:40:59.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:40:59.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:40:59.369 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:40:59.369 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","modified":"2024-02-27T07:40:53.803775Z","created":"2024-02-27T07:39:25.904582Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3301","nonce":0},{"type":"v1","addr":"172.21.15.149:6790","nonce":0}]},"addr":"172.21.15.149:6790/0","public_addr":"172.21.15.149:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-27T07:40:59.370 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[24796]: Deploying daemon mon.b on smithi195 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/2077171580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[24796]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[29527]: Deploying daemon mon.b on smithi195 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/2077171580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[29527]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:40:59 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:41:01.349 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-27T07:41:01.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T07:41:02.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:02 smithi149 ceph-mon[24796]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:02.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:02 smithi149 ceph-mon[29527]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:03.512 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:41:03.512 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","modified":"2024-02-27T07:40:53.803775Z","created":"2024-02-27T07:39:25.904582Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3301","nonce":0},{"type":"v1","addr":"172.21.15.149:6790","nonce":0}]},"addr":"172.21.15.149:6790/0","public_addr":"172.21.15.149:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-27T07:41:03.514 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-02-27T07:41:03.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:03 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/177707754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:03.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:03 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/177707754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:05.183 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-27T07:41:05.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: mon.a calling monitor election 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: mon.c calling monitor election 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: mon.b calling monitor election 2024-02-27T07:41:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: fsmap 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: mgrmap e13: y(active, since 46s) 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: overall HEALTH_OK 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:09.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: mon.a calling monitor election 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: mon.c calling monitor election 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: mon.b calling monitor election 2024-02-27T07:41:09.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: fsmap 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: mgrmap e13: y(active, since 46s) 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: overall HEALTH_OK 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:09.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:10.049 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-27T07:41:10.049 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","modified":"2024-02-27T07:41:03.812156Z","created":"2024-02-27T07:39:25.904582Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3301","nonce":0},{"type":"v1","addr":"172.21.15.149:6790","nonce":0}]},"addr":"172.21.15.149:6790/0","public_addr":"172.21.15.149:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-27T07:41:10.051 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 3 2024-02-27T07:41:10.612 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-27T07:41:10.612 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph config generate-minimal-conf 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/3023401725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:10.813 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/3023401725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:10.814 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:10.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:12.154 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: Reconfiguring mon.a (unknown last config time)... 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: Reconfiguring mon.a (unknown last config time)... 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:12.155 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:12.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:41:12.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: Reconfiguring mon.a (unknown last config time)... 2024-02-27T07:41:12.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:41:12.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:12.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:12.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:12.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[24796]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[24796]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[24796]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[29527]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[29527]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:41:12.833 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:12 smithi149 ceph-mon[29527]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:13.112 INFO:teuthology.orchestra.run.smithi149.stdout:# minimal ceph.conf for 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:41:13.112 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2024-02-27T07:41:13.112 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:41:13.112 INFO:teuthology.orchestra.run.smithi149.stdout: mon_host = [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] 2024-02-27T07:41:13.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:12 smithi195 ceph-mon[25709]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:41:13.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:12 smithi195 ceph-mon[25709]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:41:13.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:12 smithi195 ceph-mon[25709]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:14.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:13 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2266966384' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:14.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:13 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:13 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2266966384' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:14.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:13 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:13 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2266966384' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:14.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:13 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.564 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-27T07:41:14.564 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:41:14.564 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.conf 2024-02-27T07:41:14.593 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:41:14.594 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:41:14.661 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:41:14.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-02-27T07:41:14.690 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:41:14.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:41:14.757 INFO:tasks.cephadm:Adding mgr.y on smithi149 2024-02-27T07:41:14.757 INFO:tasks.cephadm:Adding mgr.x on smithi195 2024-02-27T07:41:14.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply mgr '2;smithi149=y;smithi195=x' 2024-02-27T07:41:14.823 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:14 smithi195 ceph-mon[25709]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:14.824 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.824 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:14.824 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:14.824 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:14.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[24796]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[29527]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:41:14.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:16.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:15 smithi195 ceph-mon[25709]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:41:16.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:15 smithi195 ceph-mon[25709]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:41:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:15 smithi149 ceph-mon[24796]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:41:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:15 smithi149 ceph-mon[24796]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:41:16.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:15 smithi149 ceph-mon[29527]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:41:16.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:15 smithi149 ceph-mon[29527]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:41:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:16.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:16 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:16.993 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mgr update... 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:17.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:16 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:17.607 DEBUG:teuthology.orchestra.run.smithi195:mgr.x> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service 2024-02-27T07:41:17.650 INFO:tasks.cephadm:Deploying OSDs... 2024-02-27T07:41:17.650 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:41:17.650 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-02-27T07:41:17.669 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-27T07:41:17.669 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-02-27 07:40:53.230019248 +0000 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-02-27 07:38:15.671114875 +0000 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-02-27 07:38:15.671114875 +0000 2024-02-27T07:41:17.726 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-02-27 07:38:15.671114875 +0000 2024-02-27T07:41:17.726 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-27T07:41:17.797 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-02-27T07:41:17.797 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-02-27T07:41:17.797 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000212187 s, 2.4 MB/s 2024-02-27T07:41:17.798 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-27T07:41:17.858 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-02-27 07:40:53.230019248 +0000 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-02-27 07:38:16.023107858 +0000 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-02-27 07:38:16.023107858 +0000 2024-02-27T07:41:17.915 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-02-27 07:38:16.023107858 +0000 2024-02-27T07:41:17.915 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-27T07:41:17.983 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-02-27T07:41:17.983 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-02-27T07:41:17.983 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000211003 s, 2.4 MB/s 2024-02-27T07:41:17.985 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-27T07:41:18.042 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-02-27T07:41:18.099 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-27T07:41:18.099 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:18.099 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-27T07:41:18.099 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:18.099 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:18.100 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-02-27 07:40:53.230019248 +0000 2024-02-27T07:41:18.100 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-02-27 07:38:16.360101139 +0000 2024-02-27T07:41:18.100 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-02-27 07:38:16.360101139 +0000 2024-02-27T07:41:18.100 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-02-27 07:38:16.360101139 +0000 2024-02-27T07:41:18.100 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi149=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: Saving service mgr spec with placement smithi149=y;smithi195=x;count:2 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:18.165 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: Deploying daemon mgr.x on smithi195 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[24796]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi149=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: Saving service mgr spec with placement smithi149=y;smithi195=x;count:2 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:18.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: Deploying daemon mgr.x on smithi195 2024-02-27T07:41:18.167 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:17 smithi149 ceph-mon[29527]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:18.167 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-02-27T07:41:18.167 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-02-27T07:41:18.167 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000187116 s, 2.7 MB/s 2024-02-27T07:41:18.168 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-27T07:41:18.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi149=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:18.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: Saving service mgr spec with placement smithi149=y;smithi195=x;count:2 2024-02-27T07:41:18.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: Deploying daemon mgr.x on smithi195 2024-02-27T07:41:18.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:17 smithi195 ceph-mon[25709]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:18.225 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:18.281 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-02-27 07:40:53.231019228 +0000 2024-02-27T07:41:18.282 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-02-27 07:38:16.699094381 +0000 2024-02-27T07:41:18.282 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-02-27 07:38:16.699094381 +0000 2024-02-27T07:41:18.282 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-02-27 07:38:16.699094381 +0000 2024-02-27T07:41:18.282 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-27T07:41:18.348 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-02-27T07:41:18.348 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-02-27T07:41:18.348 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000183313 s, 2.8 MB/s 2024-02-27T07:41:18.349 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-27T07:41:18.404 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:41:18.404 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-02-27T07:41:18.421 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-27T07:41:18.421 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:18.479 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-27 07:41:02.497817233 +0000 2024-02-27T07:41:18.480 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-27 07:38:15.659447152 +0000 2024-02-27T07:41:18.480 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-27 07:38:15.659447152 +0000 2024-02-27T07:41:18.480 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-02-27 07:38:15.659447152 +0000 2024-02-27T07:41:18.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-27T07:41:18.547 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-27T07:41:18.547 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-27T07:41:18.547 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000115711 s, 4.4 MB/s 2024-02-27T07:41:18.549 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-27T07:41:18.606 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:18.662 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-27 07:41:02.497817233 +0000 2024-02-27T07:41:18.663 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-27 07:38:15.996441051 +0000 2024-02-27T07:41:18.663 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-27 07:38:15.996441051 +0000 2024-02-27T07:41:18.663 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-02-27 07:38:15.996441051 +0000 2024-02-27T07:41:18.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-27T07:41:18.811 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-27T07:41:18.811 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-27T07:41:18.811 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.831e-05 s, 5.8 MB/s 2024-02-27T07:41:18.813 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-27T07:41:18.828 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-02-27T07:41:18.885 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-27T07:41:18.885 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:18.885 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-27T07:41:18.885 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:18.885 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:18.886 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-27 07:41:18.767513733 +0000 2024-02-27T07:41:18.886 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-27 07:38:16.334434931 +0000 2024-02-27T07:41:18.886 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-27 07:38:16.334434931 +0000 2024-02-27T07:41:18.886 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-02-27 07:38:16.334434931 +0000 2024-02-27T07:41:18.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-27T07:41:18.980 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-27T07:41:18.980 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-27T07:41:18.980 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000116779 s, 4.4 MB/s 2024-02-27T07:41:18.982 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-27T07:41:18.997 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-27 07:41:18.940510490 +0000 2024-02-27T07:41:19.054 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-27 07:38:16.678428702 +0000 2024-02-27T07:41:19.055 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-27 07:38:16.678428702 +0000 2024-02-27T07:41:19.055 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-02-27 07:38:16.678428702 +0000 2024-02-27T07:41:19.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-27T07:41:19.119 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:18 smithi195 systemd[1]: Starting Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:41:19.122 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-27T07:41:19.122 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-27T07:41:19.122 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.6166e-05 s, 5.3 MB/s 2024-02-27T07:41:19.123 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-27T07:41:19.178 INFO:tasks.cephadm:Deploying osd.0 on smithi149 with /dev/vg_nvme/lv_4... 2024-02-27T07:41:19.179 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-27T07:41:19.418 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 podman[26843]: 2024-02-27 07:41:19.197414019 +0000 UTC m=+0.100430359 container create 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, ceph=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, version=8, vcs-type=git, io.openshift.tags=base centos centos-stream) 2024-02-27T07:41:19.418 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 podman[26843]: 2024-02-27 07:41:19.116198155 +0000 UTC m=+0.019214495 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-27T07:41:19.418 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 podman[26843]: 2024-02-27 07:41:19.41640999 +0000 UTC m=+0.319426330 container init 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, GIT_CLEAN=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-27T07:41:19.668 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 podman[26843]: 2024-02-27 07:41:19.418961583 +0000 UTC m=+0.321977924 container start 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, release=754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, distribution-scope=public, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=) 2024-02-27T07:41:19.668 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 ceph-mgr[26899]: -- 172.21.15.195:0/489801920 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8ba988340 con 0x55c8ba960c00 2024-02-27T07:41:19.668 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 bash[26843]: 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 2024-02-27T07:41:19.668 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 systemd[1]: Started Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:41:20.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:19.936+0000 7f31b2962000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:41:20.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:20.031+0000 7f31b2962000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:20.713 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:20.714 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:20 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:20.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:20.644+0000 7f31b2962000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:41:20.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:20.732+0000 7f31b2962000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:41:20.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:20.814+0000 7f31b2962000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:41:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:41:20.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132292 s, 793 MB/s 2024-02-27T07:41:21.207 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-02-27T07:41:21.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:20.961+0000 7f31b2962000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:41:21.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:21.040+0000 7f31b2962000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:41:21.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:21.148+0000 7f31b2962000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:41:21.596 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:21.232+0000 7f31b2962000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:41:21.596 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:21.394+0000 7f31b2962000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:41:21.596 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:21.479+0000 7f31b2962000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:41:21.925 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:21 smithi149 ceph-mon[24796]: Reconfiguring mgr.y (unknown last config time)... 2024-02-27T07:41:21.925 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:21 smithi149 ceph-mon[24796]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:41:21.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:21 smithi149 ceph-mon[29527]: Reconfiguring mgr.y (unknown last config time)... 2024-02-27T07:41:21.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:21 smithi149 ceph-mon[29527]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:41:21.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-mon[25709]: Reconfiguring mgr.y (unknown last config time)... 2024-02-27T07:41:21.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:21 smithi195 ceph-mon[25709]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:41:22.187 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_4 2024-02-27T07:41:22.806 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:22 smithi149 ceph-mon[24796]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:22.806 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:22 smithi149 ceph-mon[29527]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:22.823 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:22 smithi195 ceph-mon[25709]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:22.823 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:22.501+0000 7f31b2962000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:41:22.823 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:22.632+0000 7f31b2962000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:41:22.823 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:22.820+0000 7f31b2962000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:41:23.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:22.906+0000 7f31b2962000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:41:23.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:23.040+0000 7f31b2962000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:41:23.670 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:23.364+0000 7f31b2962000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:41:23.670 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:23.582+0000 7f31b2962000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:41:23.670 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:23.667+0000 7f31b2962000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:41:23.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:41:23.769 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:23.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:41:23.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:23.836+0000 7f31b2962000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[24796]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[29527]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:24.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:24.966 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:24.769+0000 7f31b2962000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:41:24.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:24 smithi195 ceph-mon[25709]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:24.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:24.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:24.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:25.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:25.058+0000 7f31b2962000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:41:25.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:25.152+0000 7f31b2962000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:41:25.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:41:25.303+0000 7f31b2962000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: Standby manager daemon x started 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:41:25.808 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:41:25.809 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:25 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:41:25.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:25 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1756084501' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:41:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[24796]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[24796]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-27T07:41:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:41:26.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[29527]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:26.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[29527]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-27T07:41:26.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:26 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:41:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:26 smithi195 ceph-mon[25709]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:26 smithi195 ceph-mon[25709]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-27T07:41:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:26 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/38708717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]': finished 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: osdmap e5: 1 total, 0 up, 1 in 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3172106009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/38708717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]': finished 2024-02-27T07:41:27.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: osdmap e5: 1 total, 0 up, 1 in 2024-02-27T07:41:27.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:27.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:27 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3172106009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/38708717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]: dispatch 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fb57335-e582-4e40-82c3-5468f57f1fbf"}]': finished 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: osdmap e5: 1 total, 0 up, 1 in 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:27.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:27 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3172106009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:28.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:28 smithi149 ceph-mon[24796]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:28.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:28 smithi149 ceph-mon[29527]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:28.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:28 smithi195 ceph-mon[25709]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:30.812 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:30 smithi149 ceph-mon[24796]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:30.812 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:30 smithi149 ceph-mon[29527]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:30 smithi195 ceph-mon[25709]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:32 smithi149 ceph-mon[24796]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:32 smithi149 ceph-mon[29527]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:32.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:32 smithi195 ceph-mon[25709]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:33.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:41:33.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:33.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:33 smithi195 ceph-mon[25709]: Deploying daemon osd.0 on smithi149 2024-02-27T07:41:33.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:41:33.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:33.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[24796]: Deploying daemon osd.0 on smithi149 2024-02-27T07:41:33.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:41:33.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:33.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:33 smithi149 ceph-mon[29527]: Deploying daemon osd.0 on smithi149 2024-02-27T07:41:34.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:34 smithi149 ceph-mon[24796]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:34.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:34 smithi149 ceph-mon[29527]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:34.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:34 smithi195 ceph-mon[25709]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:36 smithi149 ceph-mon[24796]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:36 smithi149 ceph-mon[29527]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:36 smithi195 ceph-mon[25709]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:38.884 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[24796]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[29527]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:38.885 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:38 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:38 smithi195 ceph-mon[25709]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:38 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:38 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:38 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:38.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:38 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:40.412 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 0 on host 'smithi149' 2024-02-27T07:41:40.675 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[29527]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:40.675 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:40.675 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:40.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[24796]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:40.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:40.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:40.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:40 smithi195 ceph-mon[25709]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:40.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:40.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:41.044 DEBUG:teuthology.orchestra.run.smithi149:osd.0> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0.service 2024-02-27T07:41:41.046 INFO:tasks.cephadm:Deploying osd.1 on smithi149 with /dev/vg_nvme/lv_3... 2024-02-27T07:41:41.046 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-27T07:41:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:41 smithi149 ceph-mon[24796]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:41 smithi149 ceph-mon[24796]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:41 smithi149 ceph-mon[29527]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:41 smithi149 ceph-mon[29527]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:41 smithi195 ceph-mon[25709]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:41 smithi195 ceph-mon[25709]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: osdmap e6: 1 total, 0 up, 1 in 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[24796]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:42.664 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:42.665 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:41:42.665 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: osdmap e6: 1 total, 0 up, 1 in 2024-02-27T07:41:42.665 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:42.665 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:42.665 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-mon[29527]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:42.665 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[32383]: 2024-02-27T07:41:42.537+0000 7f148e051700 -1 osd.0 0 waiting for initial osdmap 2024-02-27T07:41:42.665 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:41:42 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[32383]: 2024-02-27T07:41:42.540+0000 7f14861e3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: osdmap e6: 1 total, 0 up, 1 in 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: from='osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:42 smithi195 ceph-mon[25709]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125044 s, 839 MB/s 2024-02-27T07:41:43.331 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[29527]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[29527]: osdmap e7: 1 total, 0 up, 1 in 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[24796]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:41:43.672 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[24796]: osdmap e7: 1 total, 0 up, 1 in 2024-02-27T07:41:43.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:43.673 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:43.929 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_3 2024-02-27T07:41:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:43 smithi195 ceph-mon[25709]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:41:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:43 smithi195 ceph-mon[25709]: osdmap e7: 1 total, 0 up, 1 in 2024-02-27T07:41:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:43 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:43 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.612 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258] boot 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: osdmap e8: 1 total, 1 up, 1 in 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.613 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258] boot 2024-02-27T07:41:44.614 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: osdmap e8: 1 total, 1 up, 1 in 2024-02-27T07:41:44.614 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:44 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:41:44.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:41:44.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T07:41:44.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:44.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: osd.0 [v2:172.21.15.149:6802/2126977258,v1:172.21.15.149:6803/2126977258] boot 2024-02-27T07:41:44.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: osdmap e8: 1 total, 1 up, 1 in 2024-02-27T07:41:44.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:44 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:41:45.800 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:45 smithi149 ceph-mon[24796]: osdmap e9: 1 total, 1 up, 1 in 2024-02-27T07:41:45.800 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:45 smithi149 ceph-mon[29527]: osdmap e9: 1 total, 1 up, 1 in 2024-02-27T07:41:45.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:45 smithi195 ceph-mon[25709]: osdmap e9: 1 total, 1 up, 1 in 2024-02-27T07:41:46.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:46.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:41:46.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:46.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:46.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:46.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:46.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:41:46.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:41:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:41:47.817 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi149 2024-02-27T07:41:47.817 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi149 to 4792M 2024-02-27T07:41:47.817 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi149 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi149 to 4792M 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:47.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:47 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:47 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi149 2024-02-27T07:41:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:47 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi149 to 4792M 2024-02-27T07:41:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:47 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:41:47.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:47 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:41:47.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:47 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:48.815 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:48 smithi149 ceph-mon[24796]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:48.816 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:48 smithi149 ceph-mon[24796]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:48.816 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:48 smithi149 ceph-mon[29527]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:48.816 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:48 smithi149 ceph-mon[29527]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:48.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:48 smithi195 ceph-mon[25709]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:41:48.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:48 smithi195 ceph-mon[25709]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]: dispatch 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]': finished 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[29527]: osdmap e10: 2 total, 1 up, 2 in 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]: dispatch 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]': finished 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[24796]: osdmap e10: 2 total, 1 up, 2 in 2024-02-27T07:41:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:49 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:41:49.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:49 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]: dispatch 2024-02-27T07:41:49.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:49 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1840214570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a8e4a89-1b35-4c2d-856c-21450b2d55e8"}]': finished 2024-02-27T07:41:49.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:49 smithi195 ceph-mon[25709]: osdmap e10: 2 total, 1 up, 2 in 2024-02-27T07:41:49.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:49 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:41:50.830 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:50 smithi149 ceph-mon[29527]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:50.830 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:50 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4069961079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:50.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:50 smithi149 ceph-mon[24796]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:50.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:50 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4069961079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:50.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:50 smithi195 ceph-mon[25709]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:50.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:50 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4069961079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:41:52.727 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:52 smithi149 ceph-mon[24796]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:52.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:52 smithi149 ceph-mon[29527]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:52.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:52 smithi195 ceph-mon[25709]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:54.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:54 smithi149 ceph-mon[24796]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:54 smithi149 ceph-mon[29527]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:54.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:54 smithi195 ceph-mon[25709]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:55.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:55 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:41:55.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:55 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:55.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:55 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:41:55.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:55 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:55.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:55 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:41:55.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:55 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:41:56.848 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:56 smithi149 ceph-mon[29527]: Deploying daemon osd.1 on smithi149 2024-02-27T07:41:56.848 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:56 smithi149 ceph-mon[29527]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:56.849 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:56 smithi149 ceph-mon[24796]: Deploying daemon osd.1 on smithi149 2024-02-27T07:41:56.849 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:56 smithi149 ceph-mon[24796]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:56.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:56 smithi195 ceph-mon[25709]: Deploying daemon osd.1 on smithi149 2024-02-27T07:41:56.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:56 smithi195 ceph-mon[25709]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:58.656 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:41:58 smithi149 ceph-mon[24796]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:58.657 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:41:58 smithi149 ceph-mon[29527]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:41:58.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:41:58 smithi195 ceph-mon[25709]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:00.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[29527]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:00.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:00.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:00.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:00.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:00.850 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[24796]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:00.850 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:00.850 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:00.850 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:00.850 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:00 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:00 smithi195 ceph-mon[25709]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:00 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:00 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:00 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:00 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:02.751 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 1 on host 'smithi149' 2024-02-27T07:42:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:02 smithi149 ceph-mon[24796]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:02.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:02 smithi149 ceph-mon[29527]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:02.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:02 smithi195 ceph-mon[25709]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:03.761 DEBUG:teuthology.orchestra.run.smithi149:osd.1> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.1.service 2024-02-27T07:42:03.764 INFO:tasks.cephadm:Deploying osd.2 on smithi149 with /dev/vg_nvme/lv_2... 2024-02-27T07:42:03.764 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[29527]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[29527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[24796]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:04.108 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:03 smithi149 ceph-mon[24796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:04.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:03 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:04.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:03 smithi195 ceph-mon[25709]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:04.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:03 smithi195 ceph-mon[25709]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:42:05.012 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:05.012 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:42:05.012 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: osdmap e11: 2 total, 1 up, 2 in 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: osdmap e11: 2 total, 1 up, 2 in 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.013 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:05.014 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[35367]: 2024-02-27T07:42:04.758+0000 7fc519bd1700 -1 osd.1 0 waiting for initial osdmap 2024-02-27T07:42:05.014 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[35367]: 2024-02-27T07:42:04.761+0000 7fc514568700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:42:05.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: osdmap e11: 2 total, 1 up, 2 in 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: from='osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:05.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:04 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[24796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[24796]: osdmap e12: 2 total, 1 up, 2 in 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[24796]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[29527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[29527]: osdmap e12: 2 total, 1 up, 2 in 2024-02-27T07:42:06.124 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.125 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.125 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:05 smithi149 ceph-mon[29527]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:06.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:05 smithi195 ceph-mon[25709]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:06.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:05 smithi195 ceph-mon[25709]: osdmap e12: 2 total, 1 up, 2 in 2024-02-27T07:42:06.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:05 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:05 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:05 smithi195 ceph-mon[25709]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128509 s, 816 MB/s 2024-02-27T07:42:06.233 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-02-27T07:42:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:42:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:42:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[24796]: osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184] boot 2024-02-27T07:42:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[24796]: osdmap e13: 2 total, 2 up, 2 in 2024-02-27T07:42:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:06.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:42:06.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:42:06.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[29527]: osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184] boot 2024-02-27T07:42:06.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[29527]: osdmap e13: 2 total, 2 up, 2 in 2024-02-27T07:42:06.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:06 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:07.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:06 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:42:07.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:06 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:42:07.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:06 smithi195 ceph-mon[25709]: osd.1 [v2:172.21.15.149:6810/3352774184,v1:172.21.15.149:6811/3352774184] boot 2024-02-27T07:42:07.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:06 smithi195 ceph-mon[25709]: osdmap e13: 2 total, 2 up, 2 in 2024-02-27T07:42:07.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:06 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:42:07.410 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_2 2024-02-27T07:42:08.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:07 smithi149 ceph-mon[29527]: osdmap e14: 2 total, 2 up, 2 in 2024-02-27T07:42:08.166 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:07 smithi149 ceph-mon[29527]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:08.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:07 smithi149 ceph-mon[24796]: osdmap e14: 2 total, 2 up, 2 in 2024-02-27T07:42:08.166 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:07 smithi149 ceph-mon[24796]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:07 smithi195 ceph-mon[25709]: osdmap e14: 2 total, 2 up, 2 in 2024-02-27T07:42:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:07 smithi195 ceph-mon[25709]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi149 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi149 to 2396M 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.450 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi149 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi149 to 2396M 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.451 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:09.452 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:09.452 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:09.452 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:09 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi149 2024-02-27T07:42:09.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi149 to 2396M 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:09.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:09 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:10.326 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[24796]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:10.326 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:10.326 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:10.326 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:10.327 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[29527]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:10.327 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:10.327 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:10.327 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:10 smithi195 ceph-mon[25709]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:11 smithi149 ceph-mon[24796]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:11.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:11 smithi149 ceph-mon[29527]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:11.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:11 smithi195 ceph-mon[25709]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:12 smithi195 ceph-mon[25709]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:12 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]: dispatch 2024-02-27T07:42:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:12 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]': finished 2024-02-27T07:42:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:12 smithi195 ceph-mon[25709]: osdmap e15: 3 total, 2 up, 3 in 2024-02-27T07:42:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:12 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[29527]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]: dispatch 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]': finished 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[29527]: osdmap e15: 3 total, 2 up, 3 in 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[24796]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]: dispatch 2024-02-27T07:42:12.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/990903939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b11afcf-4865-46e6-a6ae-9c96ab8580ae"}]': finished 2024-02-27T07:42:12.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[24796]: osdmap e15: 3 total, 2 up, 3 in 2024-02-27T07:42:12.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:12 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:13.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:13 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2582778707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:13.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:13 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2582778707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:13.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:13 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2582778707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:14.898 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:14 smithi149 ceph-mon[24796]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:14.898 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:14 smithi149 ceph-mon[29527]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:14.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:14 smithi195 ceph-mon[25709]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:16.763 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:16 smithi149 ceph-mon[29527]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:16.763 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:16 smithi149 ceph-mon[24796]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:16 smithi195 ceph-mon[25709]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:18.848 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[24796]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:18.848 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:42:18.848 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:18.848 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[29527]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:42:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:18.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:18 smithi195 ceph-mon[25709]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:18.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:42:18.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:19.602 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:19 smithi149 ceph-mon[24796]: Deploying daemon osd.2 on smithi149 2024-02-27T07:42:19.602 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:19 smithi149 ceph-mon[29527]: Deploying daemon osd.2 on smithi149 2024-02-27T07:42:19.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:19 smithi195 ceph-mon[25709]: Deploying daemon osd.2 on smithi149 2024-02-27T07:42:20.907 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:20 smithi149 ceph-mon[24796]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:20.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:20 smithi149 ceph-mon[29527]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:20 smithi195 ceph-mon[25709]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:22.791 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:22 smithi149 ceph-mon[24796]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:22.791 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:22 smithi149 ceph-mon[29527]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:22.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:22 smithi195 ceph-mon[25709]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:24.282 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:24.283 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:24.284 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:42:24.284 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:42:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:24.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:24.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:24.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:42:24.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:42:25.989 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 2 on host 'smithi149' 2024-02-27T07:42:26.694 DEBUG:teuthology.orchestra.run.smithi149:osd.2> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.2.service 2024-02-27T07:42:26.696 INFO:tasks.cephadm:Deploying osd.3 on smithi149 with /dev/vg_nvme/lv_1... 2024-02-27T07:42:26.697 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[24796]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[24796]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[24796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[29527]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:26.721 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[29527]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:26.722 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:26 smithi149 ceph-mon[29527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:26 smithi195 ceph-mon[25709]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:26 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:26.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:26 smithi195 ceph-mon[25709]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:26.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:26 smithi195 ceph-mon[25709]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: osdmap e16: 3 total, 2 up, 3 in 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[29527]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:28.138 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: osdmap e16: 3 total, 2 up, 3 in 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-mon[24796]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:28.139 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[38288]: 2024-02-27T07:42:27.992+0000 7f5816da1700 -1 osd.2 0 waiting for initial osdmap 2024-02-27T07:42:28.139 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[38288]: 2024-02-27T07:42:27.995+0000 7f5810736700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:42:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:42:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: osdmap e16: 3 total, 2 up, 3 in 2024-02-27T07:42:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:28.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: from='osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:28.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:27 smithi195 ceph-mon[25709]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:28.679 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-27T07:42:28.679 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-27T07:42:28.679 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-02-27T07:42:28.680 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-02-27T07:42:28.680 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132409 s, 792 MB/s 2024-02-27T07:42:28.680 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[29527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[29527]: osdmap e17: 3 total, 2 up, 3 in 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[24796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:29.042 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[24796]: osdmap e17: 3 total, 2 up, 3 in 2024-02-27T07:42:29.043 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.043 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:28 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:28 smithi195 ceph-mon[25709]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:28 smithi195 ceph-mon[25709]: osdmap e17: 3 total, 2 up, 3 in 2024-02-27T07:42:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:28 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:28 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:29.661 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_1 2024-02-27T07:42:30.269 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750] boot 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: osdmap e18: 3 total, 3 up, 3 in 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:29 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750] boot 2024-02-27T07:42:30.270 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: osdmap e18: 3 total, 3 up, 3 in 2024-02-27T07:42:30.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:30.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:30.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:30 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-27T07:42:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:42:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:42:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: osd.2 [v2:172.21.15.149:6818/4134394750,v1:172.21.15.149:6819/4134394750] boot 2024-02-27T07:42:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: osdmap e18: 3 total, 3 up, 3 in 2024-02-27T07:42:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:42:30.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T07:42:30.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:29 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[24796]: osdmap e19: 3 total, 3 up, 3 in 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[29527]: osdmap e19: 3 total, 3 up, 3 in 2024-02-27T07:42:31.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-27T07:42:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-27T07:42:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:31 smithi195 ceph-mon[25709]: osdmap e19: 3 total, 3 up, 3 in 2024-02-27T07:42:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-27T07:42:32.045 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39552]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:32.045 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39552]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[29527]: osdmap e20: 3 total, 3 up, 3 in 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[29527]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[24796]: osdmap e20: 3 total, 3 up, 3 in 2024-02-27T07:42:32.046 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:32 smithi149 ceph-mon[24796]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:32.295 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39552]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:32.295 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39597]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:32.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-27T07:42:32.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:32 smithi195 ceph-mon[25709]: osdmap e20: 3 total, 3 up, 3 in 2024-02-27T07:42:32.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:32 smithi195 ceph-mon[25709]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:32.575 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39597]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:32.575 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39597]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:32.835 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39667]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:32.836 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39667]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:32.836 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39667]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39686]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39686]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:42:32 smithi149 sudo[39686]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39694]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39694]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:33.095 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39694]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: osdmap e21: 3 total, 3 up, 3 in 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: osdmap e21: 3 total, 3 up, 3 in 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:33.096 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:33.097 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:33.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: osdmap e21: 3 total, 3 up, 3 in 2024-02-27T07:42:33.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:33.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:33.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:33.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:33.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:33.604 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39714]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:33.605 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39714]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:33.605 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39714]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:33.859 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39734]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:33.859 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39734]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:33.859 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39734]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39744]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39744]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39744]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39737]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39737]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:34.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:33 smithi149 sudo[39737]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:34.370 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39766]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:34.628 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi149 2024-02-27T07:42:34.628 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.628 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi149 to 1597M 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.629 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27224]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27224]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:34.630 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27224]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi149 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi149 to 1597M 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:34.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39766]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 sudo[39766]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi149 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi149 to 1597M 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:34.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:34.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:34.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27227]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T07:42:34.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27227]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T07:42:34.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:34 smithi195 sudo[27227]: pam_unix(sudo:session): session closed for user root 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:35.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]': finished 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: osdmap e22: 4 total, 3 up, 4 in 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:35.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]: dispatch 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]': finished 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: osdmap e22: 4 total, 3 up, 4 in 2024-02-27T07:42:35.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:35.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]: dispatch 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2068834475' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89097d49-cef0-48d1-a00a-e303f6cc9e62"}]': finished 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: osdmap e22: 4 total, 3 up, 4 in 2024-02-27T07:42:35.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:36 smithi149 ceph-mon[29527]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:36 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3060097681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:36 smithi149 ceph-mon[24796]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:36 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3060097681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:36.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:36 smithi195 ceph-mon[25709]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:36.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:36 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3060097681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:38.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:38 smithi149 ceph-mon[24796]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 285 KiB/s wr, 21 op/s 2024-02-27T07:42:38.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:38 smithi149 ceph-mon[29527]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 285 KiB/s wr, 21 op/s 2024-02-27T07:42:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:38 smithi195 ceph-mon[25709]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 285 KiB/s wr, 21 op/s 2024-02-27T07:42:40.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:40 smithi149 ceph-mon[24796]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 209 KiB/s wr, 15 op/s 2024-02-27T07:42:40.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:40 smithi149 ceph-mon[29527]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 209 KiB/s wr, 15 op/s 2024-02-27T07:42:40.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:40 smithi195 ceph-mon[25709]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.1 KiB/s rd, 209 KiB/s wr, 15 op/s 2024-02-27T07:42:41.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:41 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:42:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:41 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:41 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:42:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:41 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:41 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:42:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:41 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:42.845 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:42 smithi149 ceph-mon[24796]: Deploying daemon osd.3 on smithi149 2024-02-27T07:42:42.845 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:42 smithi149 ceph-mon[24796]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 165 KiB/s wr, 12 op/s 2024-02-27T07:42:42.845 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:42 smithi149 ceph-mon[29527]: Deploying daemon osd.3 on smithi149 2024-02-27T07:42:42.846 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:42 smithi149 ceph-mon[29527]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 165 KiB/s wr, 12 op/s 2024-02-27T07:42:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:42 smithi195 ceph-mon[25709]: Deploying daemon osd.3 on smithi149 2024-02-27T07:42:42.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:42 smithi195 ceph-mon[25709]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 165 KiB/s wr, 12 op/s 2024-02-27T07:42:44.757 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:44 smithi149 ceph-mon[24796]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-02-27T07:42:44.757 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:44 smithi149 ceph-mon[29527]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-02-27T07:42:44.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:44 smithi195 ceph-mon[25709]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-02-27T07:42:46.592 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[29527]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 152 KiB/s wr, 11 op/s 2024-02-27T07:42:46.592 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:46.592 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:46.592 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:46.592 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:46.593 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[24796]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 152 KiB/s wr, 11 op/s 2024-02-27T07:42:46.593 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:46.593 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:46.593 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:46.593 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:46 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:46 smithi195 ceph-mon[25709]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 152 KiB/s wr, 11 op/s 2024-02-27T07:42:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:46 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:48.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:48 smithi149 ceph-mon[24796]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 130 KiB/s wr, 9 op/s 2024-02-27T07:42:48.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:48 smithi149 ceph-mon[29527]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 130 KiB/s wr, 9 op/s 2024-02-27T07:42:48.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:48 smithi195 ceph-mon[25709]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 130 KiB/s wr, 9 op/s 2024-02-27T07:42:49.094 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 3 on host 'smithi149' 2024-02-27T07:42:50.039 DEBUG:teuthology.orchestra.run.smithi149:osd.3> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.3.service 2024-02-27T07:42:50.042 INFO:tasks.cephadm:Deploying osd.4 on smithi195 with /dev/vg_nvme/lv_4... 2024-02-27T07:42:50.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-27T07:42:50.300 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:50.300 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[29527]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.300 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[29527]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.300 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[29527]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:50.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:50.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[24796]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[24796]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:50 smithi149 ceph-mon[24796]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:50.331 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:50 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:50.332 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:50 smithi195 ceph-mon[25709]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.332 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:50 smithi195 ceph-mon[25709]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:42:50.332 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:50 smithi195 ceph-mon[25709]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:51.320 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:42:51.320 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: osdmap e23: 4 total, 3 up, 4 in 2024-02-27T07:42:51.320 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: osdmap e23: 4 total, 3 up, 4 in 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:51.321 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[41253]: 2024-02-27T07:42:51.103+0000 7fa2f8010700 -1 osd.3 0 waiting for initial osdmap 2024-02-27T07:42:51.322 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:42:51 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[41253]: 2024-02-27T07:42:51.107+0000 7fa2f41aa700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:42:51.431 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:42:51.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: osdmap e23: 4 total, 3 up, 4 in 2024-02-27T07:42:51.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:51.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: from='osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:42:51.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:51 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:52.009 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-27T07:42:52.009 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-27T07:42:52.009 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-27T07:42:52.010 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-27T07:42:52.010 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00847648 s, 1.2 GB/s 2024-02-27T07:42:52.010 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[29527]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[29527]: osdmap e24: 4 total, 3 up, 4 in 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[29527]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[24796]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[24796]: osdmap e24: 4 total, 3 up, 4 in 2024-02-27T07:42:52.217 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.218 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.218 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:52 smithi149 ceph-mon[24796]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:52.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:52 smithi195 ceph-mon[25709]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-02-27T07:42:52.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:52 smithi195 ceph-mon[25709]: osdmap e24: 4 total, 3 up, 4 in 2024-02-27T07:42:52.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:52 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:52 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:52.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:52 smithi195 ceph-mon[25709]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T07:42:52.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_4 2024-02-27T07:42:53.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:53 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:42:53.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:53 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:42:53.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:53 smithi195 ceph-mon[25709]: osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260] boot 2024-02-27T07:42:53.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:53 smithi195 ceph-mon[25709]: osdmap e25: 4 total, 4 up, 4 in 2024-02-27T07:42:53.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:53 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[29527]: osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260] boot 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[29527]: osdmap e25: 4 total, 4 up, 4 in 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[24796]: osd.3 [v2:172.21.15.149:6826/3598442260,v1:172.21.15.149:6827/3598442260] boot 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[24796]: osdmap e25: 4 total, 4 up, 4 in 2024-02-27T07:42:53.271 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:53 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:42:54.177 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: osdmap e26: 4 total, 4 up, 4 in 2024-02-27T07:42:54.177 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:54.177 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi149 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi149 to 1198M 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:54.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: osdmap e26: 4 total, 4 up, 4 in 2024-02-27T07:42:54.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:54.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi149 2024-02-27T07:42:54.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi149 to 1198M 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: osdmap e26: 4 total, 4 up, 4 in 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi149 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi149 to 1198M 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:42:54.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:42:55.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:55 smithi195 ceph-mon[25709]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:55.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:55 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:55.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:55 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:55.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:55 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[29527]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:55.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[24796]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:42:55.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:42:55.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:42:55.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:55 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:42:56.188 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:56 smithi195 ceph-mon[25709]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:56 smithi149 ceph-mon[29527]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:56.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:56 smithi149 ceph-mon[24796]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/1964272952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]': finished 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[24796]: osdmap e27: 5 total, 4 up, 5 in 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:42:57.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/1964272952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]': finished 2024-02-27T07:42:57.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[29527]: osdmap e27: 5 total, 4 up, 5 in 2024-02-27T07:42:57.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:57 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:42:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:57 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/1964272952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:57 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]: dispatch 2024-02-27T07:42:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:57 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3668d9a-3e64-4913-8fd4-563c9c996e8e"}]': finished 2024-02-27T07:42:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:57 smithi195 ceph-mon[25709]: osdmap e27: 5 total, 4 up, 5 in 2024-02-27T07:42:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:57 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:42:58.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:58 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/1377514279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:58.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:42:58 smithi149 ceph-mon[24796]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:58 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/1377514279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:42:58 smithi149 ceph-mon[29527]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:42:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:58 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/1377514279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:42:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:42:58 smithi195 ceph-mon[25709]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:00.877 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:00 smithi195 ceph-mon[25709]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:00.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:00 smithi149 ceph-mon[24796]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:00.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:00 smithi149 ceph-mon[29527]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:02.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:02 smithi195 ceph-mon[25709]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:02 smithi149 ceph-mon[29527]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:02.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:02 smithi149 ceph-mon[24796]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:03.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:03 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:43:03.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:03 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:03.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:03 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:43:03.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:03 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:03.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:03 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:43:03.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:03 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:04.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:04 smithi195 ceph-mon[25709]: Deploying daemon osd.4 on smithi195 2024-02-27T07:43:04.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:04 smithi195 ceph-mon[25709]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:04.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:04 smithi149 ceph-mon[24796]: Deploying daemon osd.4 on smithi195 2024-02-27T07:43:04.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:04 smithi149 ceph-mon[24796]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:04.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:04 smithi149 ceph-mon[29527]: Deploying daemon osd.4 on smithi195 2024-02-27T07:43:04.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:04 smithi149 ceph-mon[29527]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:06.856 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:06 smithi195 ceph-mon[25709]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:06.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:06 smithi149 ceph-mon[29527]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:06.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:06 smithi149 ceph-mon[24796]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:08.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:08 smithi195 ceph-mon[25709]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:08.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:08 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:08.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:08 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:08.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:08 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:08.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:08 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[24796]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[29527]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:09.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:09.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:08 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:10.726 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:10 smithi195 ceph-mon[25709]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:10.726 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:10.726 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:10 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:10.756 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 4 on host 'smithi195' 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[24796]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[29527]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:11.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:10 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:11.347 DEBUG:teuthology.orchestra.run.smithi195:osd.4> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.4.service 2024-02-27T07:43:11.349 INFO:tasks.cephadm:Deploying osd.5 on smithi195 with /dev/vg_nvme/lv_3... 2024-02-27T07:43:11.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-27T07:43:11.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:11.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:11.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:11.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:11.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:11.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:11 smithi195 ceph-mon[25709]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[24796]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:12.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:12.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:12.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:12.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:12.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:12.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:11 smithi149 ceph-mon[29527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: osdmap e28: 5 total, 4 up, 5 in 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:13.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-mon[25709]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:13.105 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[29151]: 2024-02-27T07:43:12.765+0000 7f254451f700 -1 osd.4 0 waiting for initial osdmap 2024-02-27T07:43:13.106 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:43:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[29151]: 2024-02-27T07:43:12.769+0000 7f253feb8700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: osdmap e28: 5 total, 4 up, 5 in 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[29527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:43:13.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: osdmap e28: 5 total, 4 up, 5 in 2024-02-27T07:43:13.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:13.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: from='osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:13.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:12 smithi149 ceph-mon[24796]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:14.045 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:13 smithi195 ceph-mon[25709]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:14.045 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:13 smithi195 ceph-mon[25709]: osdmap e29: 5 total, 4 up, 5 in 2024-02-27T07:43:14.045 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:13 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.045 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:13 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[29527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[29527]: osdmap e29: 5 total, 4 up, 5 in 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[24796]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[24796]: osdmap e29: 5 total, 4 up, 5 in 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:13 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00947119 s, 1.1 GB/s 2024-02-27T07:43:14.662 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-27T07:43:14.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:43:14.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:43:14.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:14.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:14.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467] boot 2024-02-27T07:43:14.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: osdmap e30: 5 total, 5 up, 5 in 2024-02-27T07:43:14.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:14 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467] boot 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: osdmap e30: 5 total, 5 up, 5 in 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T07:43:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: osd.4 [v2:172.21.15.195:6800/1574773467,v1:172.21.15.195:6801/1574773467] boot 2024-02-27T07:43:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: osdmap e30: 5 total, 5 up, 5 in 2024-02-27T07:43:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:14 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:43:15.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_3 2024-02-27T07:43:15.872 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: osdmap e31: 5 total, 5 up, 5 in 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi195 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi195 to 17088M 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:15.873 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:15 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: osdmap e31: 5 total, 5 up, 5 in 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi195 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi195 to 17088M 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: osdmap e31: 5 total, 5 up, 5 in 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi195 2024-02-27T07:43:16.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi195 to 17088M 2024-02-27T07:43:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:16.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:15 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:17.071 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:16 smithi195 ceph-mon[25709]: osdmap e32: 5 total, 5 up, 5 in 2024-02-27T07:43:17.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:16 smithi149 ceph-mon[24796]: osdmap e32: 5 total, 5 up, 5 in 2024-02-27T07:43:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:16 smithi149 ceph-mon[29527]: osdmap e32: 5 total, 5 up, 5 in 2024-02-27T07:43:17.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: osdmap e33: 5 total, 5 up, 5 in 2024-02-27T07:43:17.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:17.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:17.994 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:17.994 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:17.994 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:17 smithi195 ceph-mon[25709]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: osdmap e33: 5 total, 5 up, 5 in 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[29527]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: osdmap e33: 5 total, 5 up, 5 in 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:18.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:18.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:18.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:17 smithi149 ceph-mon[24796]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-27T07:43:18.847 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:18 smithi195 ceph-mon[25709]: osdmap e34: 5 total, 5 up, 5 in 2024-02-27T07:43:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:18 smithi149 ceph-mon[24796]: osdmap e34: 5 total, 5 up, 5 in 2024-02-27T07:43:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:18 smithi149 ceph-mon[29527]: osdmap e34: 5 total, 5 up, 5 in 2024-02-27T07:43:20.078 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/2168437800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]': finished 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: osdmap e35: 6 total, 5 up, 6 in 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/2168437800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]': finished 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: osdmap e35: 6 total, 5 up, 6 in 2024-02-27T07:43:20.079 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:19 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/2168437800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]: dispatch 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6b4d99-1f81-49ff-823e-535465189f46"}]': finished 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: osdmap e35: 6 total, 5 up, 6 in 2024-02-27T07:43:20.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:19 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:21.094 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:20 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/1138289052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:21.094 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:20 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/1138289052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:21.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:20 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/1138289052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:22.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:21 smithi149 ceph-mon[29527]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-27T07:43:22.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:21 smithi149 ceph-mon[24796]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-27T07:43:22.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:21 smithi195 ceph-mon[25709]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-27T07:43:23.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:43:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:43:23.523+0000 7f98ab1ab700 -1 calc_pg_upmaps abort due to max <= 0 2024-02-27T07:43:24.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:24 smithi195 ceph-mon[25709]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:24.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:43:24.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[24796]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[29527]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:43:24.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:43:26.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:26 smithi195 ceph-mon[25709]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:26.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:26 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:43:26.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:26 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:26.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:26 smithi195 ceph-mon[25709]: Deploying daemon osd.5 on smithi195 2024-02-27T07:43:26.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[24796]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[24796]: Deploying daemon osd.5 on smithi195 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[29527]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:26.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:26 smithi149 ceph-mon[29527]: Deploying daemon osd.5 on smithi195 2024-02-27T07:43:28.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:28 smithi195 ceph-mon[25709]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:28 smithi149 ceph-mon[24796]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:28.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:28 smithi149 ceph-mon[29527]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:30 smithi149 ceph-mon[24796]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:30.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:30 smithi149 ceph-mon[29527]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:30 smithi195 ceph-mon[25709]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:31.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:31.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:31 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:31.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:31.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:31.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:31.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:31 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:32.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:32 smithi195 ceph-mon[25709]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:32 smithi149 ceph-mon[24796]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:32 smithi149 ceph-mon[29527]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:33.924 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 5 on host 'smithi195' 2024-02-27T07:43:34.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:34.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:34.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:34.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:34 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:34.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:34.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:34 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:34.553 DEBUG:teuthology.orchestra.run.smithi195:osd.5> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.5.service 2024-02-27T07:43:34.555 INFO:tasks.cephadm:Deploying osd.6 on smithi195 with /dev/vg_nvme/lv_2... 2024-02-27T07:43:34.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[29527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[29527]: osdmap e36: 6 total, 5 up, 6 in 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[29527]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[29527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[24796]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[24796]: osdmap e36: 6 total, 5 up, 6 in 2024-02-27T07:43:35.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:35.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[24796]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:35 smithi149 ceph-mon[24796]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-mon[25709]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:43:35.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-mon[25709]: osdmap e36: 6 total, 5 up, 6 in 2024-02-27T07:43:35.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:35.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-mon[25709]: from='osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-mon[25709]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:35.466 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[32031]: 2024-02-27T07:43:35.177+0000 7f4308322700 -1 osd.5 0 waiting for initial osdmap 2024-02-27T07:43:35.466 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:43:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[32031]: 2024-02-27T07:43:35.181+0000 7f4301cb7700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:43:36.200 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:36 smithi195 ceph-mon[25709]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:36.200 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:36 smithi195 ceph-mon[25709]: osdmap e37: 6 total, 5 up, 6 in 2024-02-27T07:43:36.200 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:36 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.200 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:36 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.200 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:36 smithi195 ceph-mon[25709]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[29527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[29527]: osdmap e37: 6 total, 5 up, 6 in 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[29527]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[24796]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[24796]: osdmap e37: 6 total, 5 up, 6 in 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:36.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:36 smithi149 ceph-mon[24796]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T07:43:37.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:43:37.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:43:37.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:37.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279] boot 2024-02-27T07:43:37.401 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: osdmap e38: 6 total, 6 up, 6 in 2024-02-27T07:43:37.401 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:37 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00941822 s, 1.1 GB/s 2024-02-27T07:43:37.517 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279] boot 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: osdmap e38: 6 total, 6 up, 6 in 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:43:37.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:37.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: osd.5 [v2:172.21.15.195:6808/4162967279,v1:172.21.15.195:6809/4162967279] boot 2024-02-27T07:43:37.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: osdmap e38: 6 total, 6 up, 6 in 2024-02-27T07:43:37.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:37 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:43:38.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_2 2024-02-27T07:43:38.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:38 smithi195 ceph-mon[25709]: osdmap e39: 6 total, 6 up, 6 in 2024-02-27T07:43:38.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:38 smithi195 ceph-mon[25709]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:38.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:38 smithi149 ceph-mon[24796]: osdmap e39: 6 total, 6 up, 6 in 2024-02-27T07:43:38.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:38 smithi149 ceph-mon[24796]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:38.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:38 smithi149 ceph-mon[29527]: osdmap e39: 6 total, 6 up, 6 in 2024-02-27T07:43:38.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:38 smithi149 ceph-mon[29527]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:39.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:39 smithi195 ceph-mon[25709]: osdmap e40: 6 total, 6 up, 6 in 2024-02-27T07:43:39.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:39 smithi149 ceph-mon[24796]: osdmap e40: 6 total, 6 up, 6 in 2024-02-27T07:43:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:39 smithi149 ceph-mon[29527]: osdmap e40: 6 total, 6 up, 6 in 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: osdmap e41: 6 total, 6 up, 6 in 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi195 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi195 to 8544M 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:40 smithi195 ceph-mon[25709]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:40.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: osdmap e41: 6 total, 6 up, 6 in 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi195 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi195 to 8544M 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[29527]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: osdmap e41: 6 total, 6 up, 6 in 2024-02-27T07:43:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi195 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi195 to 8544M 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:40 smithi149 ceph-mon[24796]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:41.354 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:41 smithi195 ceph-mon[25709]: osdmap e42: 6 total, 6 up, 6 in 2024-02-27T07:43:41.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:41 smithi195 ceph-mon[25709]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:41.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:41 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:41.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:41 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:41.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:41 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[24796]: osdmap e42: 6 total, 6 up, 6 in 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[24796]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[29527]: osdmap e42: 6 total, 6 up, 6 in 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[29527]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:43:41.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:41 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:42.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:42 smithi149 ceph-mon[29527]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:42 smithi149 ceph-mon[24796]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:42.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:42 smithi195 ceph-mon[25709]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/2401119851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]': finished 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: osdmap e43: 7 total, 6 up, 7 in 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/409663993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/2401119851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]': finished 2024-02-27T07:43:43.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: osdmap e43: 7 total, 6 up, 7 in 2024-02-27T07:43:43.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:43.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:43 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/409663993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/2401119851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]: dispatch 2024-02-27T07:43:43.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bf5822-e0f1-495c-b54e-6a111eef979c"}]': finished 2024-02-27T07:43:43.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: osdmap e43: 7 total, 6 up, 7 in 2024-02-27T07:43:43.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:43.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:43 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/409663993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:43:44.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:44 smithi149 ceph-mon[29527]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:44.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:44 smithi149 ceph-mon[24796]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:44.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:44 smithi195 ceph-mon[25709]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:46.816 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:46 smithi195 ceph-mon[25709]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:46.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:46 smithi149 ceph-mon[29527]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:46.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:46 smithi149 ceph-mon[24796]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:48.653 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:48 smithi195 ceph-mon[25709]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:48.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:48 smithi149 ceph-mon[24796]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:48.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:48 smithi149 ceph-mon[29527]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:49.698 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:49 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:43:49.698 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:49 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:49 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:43:49.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:49 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:49 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:43:49.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:49 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:50.828 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:50 smithi149 ceph-mon[29527]: Deploying daemon osd.6 on smithi195 2024-02-27T07:43:50.829 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:50 smithi149 ceph-mon[29527]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:50.829 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:50 smithi149 ceph-mon[24796]: Deploying daemon osd.6 on smithi195 2024-02-27T07:43:50.829 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:50 smithi149 ceph-mon[24796]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:50.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:50 smithi195 ceph-mon[25709]: Deploying daemon osd.6 on smithi195 2024-02-27T07:43:50.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:50 smithi195 ceph-mon[25709]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:52.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:52 smithi195 ceph-mon[25709]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:52.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:52 smithi149 ceph-mon[24796]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:52.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:52 smithi149 ceph-mon[29527]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:54 smithi195 ceph-mon[25709]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:54.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:54 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:54.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[24796]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[29527]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:43:54.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:54 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:43:56.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:56 smithi195 ceph-mon[25709]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:56.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:56 smithi149 ceph-mon[29527]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:56.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:56 smithi149 ceph-mon[24796]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:57.570 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 6 on host 'smithi195' 2024-02-27T07:43:57.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:57 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:57.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:57 smithi195 ceph-mon[25709]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:57.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:57 smithi195 ceph-mon[25709]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:57.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:57 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:58.168 DEBUG:teuthology.orchestra.run.smithi195:osd.6> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6.service 2024-02-27T07:43:58.171 INFO:tasks.cephadm:Deploying osd.7 on smithi195 with /dev/vg_nvme/lv_1... 2024-02-27T07:43:58.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-27T07:43:58.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[29527]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[29527]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[24796]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[24796]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:43:58.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:57 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:43:58.952 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:58.952 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:43:58.952 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: osdmap e44: 7 total, 6 up, 7 in 2024-02-27T07:43:58.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:58.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:58.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-mon[25709]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:58.953 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[34912]: 2024-02-27T07:43:58.706+0000 7f789cdcc700 -1 osd.6 0 waiting for initial osdmap 2024-02-27T07:43:58.953 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:43:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[34912]: 2024-02-27T07:43:58.712+0000 7f7897763700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:43:59.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: osdmap e44: 7 total, 6 up, 7 in 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[29527]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: osdmap e44: 7 total, 6 up, 7 in 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: from='osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:59.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:58 smithi149 ceph-mon[24796]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:43:59.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:59 smithi195 ceph-mon[25709]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:43:59.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:59 smithi195 ceph-mon[25709]: osdmap e45: 7 total, 6 up, 7 in 2024-02-27T07:43:59.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:59 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:43:59.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:43:59 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:00.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[24796]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[24796]: osdmap e45: 7 total, 6 up, 7 in 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[29527]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[29527]: osdmap e45: 7 total, 6 up, 7 in 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:00.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:43:59 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:44:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:44:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:44:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124] boot 2024-02-27T07:44:00.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: osdmap e46: 7 total, 7 up, 7 in 2024-02-27T07:44:00.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:00 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:01.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:44:01.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124] boot 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: osdmap e46: 7 total, 7 up, 7 in 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: osd.6 [v2:172.21.15.195:6816/822039124,v1:172.21.15.195:6817/822039124] boot 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: osdmap e46: 7 total, 7 up, 7 in 2024-02-27T07:44:01.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:00 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:01.664 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-27T07:44:01.665 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-27T07:44:01.665 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-27T07:44:01.665 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-27T07:44:01.665 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00967246 s, 1.1 GB/s 2024-02-27T07:44:01.665 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-27T07:44:01.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:01 smithi195 ceph-mon[25709]: osdmap e47: 7 total, 7 up, 7 in 2024-02-27T07:44:02.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:01 smithi149 ceph-mon[29527]: osdmap e47: 7 total, 7 up, 7 in 2024-02-27T07:44:02.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:01 smithi149 ceph-mon[24796]: osdmap e47: 7 total, 7 up, 7 in 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: osdmap e48: 7 total, 7 up, 7 in 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:02.730 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: osdmap e48: 7 total, 7 up, 7 in 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.731 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:02.732 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:02.732 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:02.732 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.732 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[24796]: osdmap e49: 7 total, 7 up, 7 in 2024-02-27T07:44:02.832 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: osdmap e48: 7 total, 7 up, 7 in 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:02.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:02 smithi195 ceph-mon[25709]: osdmap e49: 7 total, 7 up, 7 in 2024-02-27T07:44:02.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_1 2024-02-27T07:44:03.048 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:02 smithi149 ceph-mon[29527]: osdmap e49: 7 total, 7 up, 7 in 2024-02-27T07:44:03.989 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:03 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi195 2024-02-27T07:44:03.989 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:03 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi195 to 5696M 2024-02-27T07:44:03.989 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:03 smithi195 ceph-mon[25709]: osdmap e50: 7 total, 7 up, 7 in 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi195 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi195 to 5696M 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[24796]: osdmap e50: 7 total, 7 up, 7 in 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi195 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi195 to 5696M 2024-02-27T07:44:04.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:03 smithi149 ceph-mon[29527]: osdmap e50: 7 total, 7 up, 7 in 2024-02-27T07:44:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:04 smithi195 ceph-mon[25709]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:04 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:44:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:04 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:44:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:04 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:05.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[24796]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[29527]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T07:44:05.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:04 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:06.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:05 smithi195 ceph-mon[25709]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:06.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:05 smithi149 ceph-mon[24796]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:06.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:05 smithi149 ceph-mon[29527]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:07.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:06 smithi149 ceph-mon[24796]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:07.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:06 smithi149 ceph-mon[29527]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:07.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:06 smithi195 ceph-mon[25709]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/1964376235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]': finished 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: osdmap e51: 8 total, 7 up, 8 in 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/2604513251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/1964376235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]': finished 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: osdmap e51: 8 total, 7 up, 8 in 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:08.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:07 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/2604513251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/1964376235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]: dispatch 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6af7cb6c-45bf-4b4b-804a-4fee7e774e20"}]': finished 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: osdmap e51: 8 total, 7 up, 8 in 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:07 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/2604513251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T07:44:09.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:08 smithi149 ceph-mon[24796]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-27T07:44:09.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:08 smithi149 ceph-mon[29527]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-27T07:44:09.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:08 smithi195 ceph-mon[25709]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-27T07:44:10.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:10 smithi195 ceph-mon[25709]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-27T07:44:11.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:10 smithi149 ceph-mon[24796]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-27T07:44:11.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:10 smithi149 ceph-mon[29527]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-27T07:44:12.902 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:12 smithi195 ceph-mon[25709]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T07:44:13.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:12 smithi149 ceph-mon[29527]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T07:44:13.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:12 smithi149 ceph-mon[24796]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T07:44:13.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:13 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:44:13.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:13 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:14.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:13 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:44:14.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:13 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:13 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:44:14.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:13 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:15.067 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:14 smithi195 ceph-mon[25709]: Deploying daemon osd.7 on smithi195 2024-02-27T07:44:15.067 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:14 smithi195 ceph-mon[25709]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:14 smithi149 ceph-mon[24796]: Deploying daemon osd.7 on smithi195 2024-02-27T07:44:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:14 smithi149 ceph-mon[24796]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:14 smithi149 ceph-mon[29527]: Deploying daemon osd.7 on smithi195 2024-02-27T07:44:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:14 smithi149 ceph-mon[29527]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:16.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:16 smithi195 ceph-mon[25709]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:17.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:16 smithi149 ceph-mon[24796]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:17.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:16 smithi149 ceph-mon[29527]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T07:44:18.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:18 smithi195 ceph-mon[25709]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:18.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:18.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:18.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:18.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:18 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[29527]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[24796]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:19.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:18 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:20.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:20 smithi195 ceph-mon[25709]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:21.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:20 smithi149 ceph-mon[24796]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:21.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:20 smithi149 ceph-mon[29527]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:21.338 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 7 on host 'smithi195' 2024-02-27T07:44:21.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:21 smithi195 ceph-mon[25709]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:44:21.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:21 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:22.003 DEBUG:teuthology.orchestra.run.smithi195:osd.7> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.7.service 2024-02-27T07:44:22.006 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-27T07:44:22.006 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd stat -f json 2024-02-27T07:44:22.036 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:21 smithi149 ceph-mon[29527]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:44:22.036 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:21 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:22.036 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:21 smithi149 ceph-mon[24796]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:44:22.037 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:21 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:22.868 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:22.868 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:44:22.868 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: osdmap e52: 8 total, 7 up, 8 in 2024-02-27T07:44:22.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:22.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:44:22.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:22.869 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[37907]: 2024-02-27T07:44:22.774+0000 7f7ebc9a6700 -1 osd.7 0 waiting for initial osdmap 2024-02-27T07:44:22.869 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:44:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[37907]: 2024-02-27T07:44:22.783+0000 7f7eb7b3e700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:44:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: osdmap e52: 8 total, 7 up, 8 in 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: osdmap e52: 8 total, 7 up, 8 in 2024-02-27T07:44:22.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:22.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:44:22.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:22 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: osdmap e53: 8 total, 7 up, 8 in 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:44:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: osdmap e53: 8 total, 7 up, 8 in 2024-02-27T07:44:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:23 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: from='osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-27T07:44:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: osdmap e53: 8 total, 7 up, 8 in 2024-02-27T07:44:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:24.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:23 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:24.341 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: purged_snaps scrub starts 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: purged_snaps scrub ok 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492] boot 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: osdmap e54: 8 total, 8 up, 8 in 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.853 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:24 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3951862543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: purged_snaps scrub starts 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: purged_snaps scrub ok 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492] boot 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: osdmap e54: 8 total, 8 up, 8 in 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3951862543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-27T07:44:24.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: purged_snaps scrub starts 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: purged_snaps scrub ok 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: osd.7 [v2:172.21.15.195:6824/2645959492,v1:172.21.15.195:6825/2645959492] boot 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: osdmap e54: 8 total, 8 up, 8 in 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:24.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:24 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3951862543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-27T07:44:25.052 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1709019863,"num_in_osds":8,"osd_in_since":1709019847,"num_remapped_pgs":1} 2024-02-27T07:44:25.053 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T07:44:25.320 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: osdmap e55: 8 total, 8 up, 8 in 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: Detected new or changed devices on smithi195 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi195 to 4272M 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[24796]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: osdmap e55: 8 total, 8 up, 8 in 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: Detected new or changed devices on smithi195 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi195 to 4272M 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:25 smithi149 ceph-mon[29527]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: osdmap e55: 8 total, 8 up, 8 in 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: Detected new or changed devices on smithi195 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi195 to 4272M 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:26.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:26.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:26.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:26.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:25 smithi195 ceph-mon[25709]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:26.820 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:26 smithi149 ceph-mon[24796]: osdmap e56: 8 total, 8 up, 8 in 2024-02-27T07:44:26.821 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:26 smithi149 ceph-mon[29527]: osdmap e56: 8 total, 8 up, 8 in 2024-02-27T07:44:27.190 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:44:27.190 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":57,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","created":"2024-02-27T07:39:28.765884+0000","modified":"2024-02-27T07:44:26.795546+0000","last_up_change":"2024-02-27T07:44:23.774911+0000","last_in_change":"2024-02-27T07:44:07.023608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T07:42:29.818441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1fb57335-e582-4e40-82c3-5468f57f1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6803","nonce":2126977258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6805","nonce":2126977258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6809","nonce":2126977258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6807","nonce":2126977258}]},"public_addr":"172.21.15.149:6803/2126977258","cluster_addr":"172.21.15.149:6805/2126977258","heartbeat_back_addr":"172.21.15.149:6809/2126977258","heartbeat_front_addr":"172.21.15.149:6807/2126977258","state":["exists","up"]},{"osd":1,"uuid":"0a8e4a89-1b35-4c2d-856c-21450b2d55e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6811","nonce":3352774184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6813","nonce":3352774184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6817","nonce":3352774184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6815","nonce":3352774184}]},"public_addr":"172.21.15.149:6811/3352774184","cluster_addr":"172.21.15.149:6813/3352774184","heartbeat_back_addr":"172.21.15.149:6817/3352774184","heartbeat_front_addr":"172.21.15.149:6815/3352774184","state":["exists","up"]},{"osd":2,"uuid":"9b11afcf-4865-46e6-a6ae-9c96ab8580ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6819","nonce":4134394750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6821","nonce":4134394750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6825","nonce":4134394750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6823","nonce":4134394750}]},"public_addr":"172.21.15.149:6819/4134394750","cluster_addr":"172.21.15.149:6821/4134394750","heartbeat_back_addr":"172.21.15.149:6825/4134394750","heartbeat_front_addr":"172.21.15.149:6823/4134394750","state":["exists","up"]},{"osd":3,"uuid":"89097d49-cef0-48d1-a00a-e303f6cc9e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6827","nonce":3598442260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6829","nonce":3598442260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6833","nonce":3598442260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6831","nonce":3598442260}]},"public_addr":"172.21.15.149:6827/3598442260","cluster_addr":"172.21.15.149:6829/3598442260","heartbeat_back_addr":"172.21.15.149:6833/3598442260","heartbeat_front_addr":"172.21.15.149:6831/3598442260","state":["exists","up"]},{"osd":4,"uuid":"e3668d9a-3e64-4913-8fd4-563c9c996e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6801","nonce":1574773467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6803","nonce":1574773467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6807","nonce":1574773467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6805","nonce":1574773467}]},"public_addr":"172.21.15.195:6801/1574773467","cluster_addr":"172.21.15.195:6803/1574773467","heartbeat_back_addr":"172.21.15.195:6807/1574773467","heartbeat_front_addr":"172.21.15.195:6805/1574773467","state":["exists","up"]},{"osd":5,"uuid":"5f6b4d99-1f81-49ff-823e-535465189f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6809","nonce":4162967279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6811","nonce":4162967279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6815","nonce":4162967279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6813","nonce":4162967279}]},"public_addr":"172.21.15.195:6809/4162967279","cluster_addr":"172.21.15.195:6811/4162967279","heartbeat_back_addr":"172.21.15.195:6815/4162967279","heartbeat_front_addr":"172.21.15.195:6813/4162967279","state":["exists","up"]},{"osd":6,"uuid":"27bf5822-e0f1-495c-b54e-6a111eef979c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6817","nonce":822039124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6819","nonce":822039124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6823","nonce":822039124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6821","nonce":822039124}]},"public_addr":"172.21.15.195:6817/822039124","cluster_addr":"172.21.15.195:6819/822039124","heartbeat_back_addr":"172.21.15.195:6823/822039124","heartbeat_front_addr":"172.21.15.195:6821/822039124","state":["exists","up"]},{"osd":7,"uuid":"6af7cb6c-45bf-4b4b-804a-4fee7e774e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6825","nonce":2645959492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6827","nonce":2645959492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6831","nonce":2645959492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6829","nonce":2645959492}]},"public_addr":"172.21.15.195:6825/2645959492","cluster_addr":"172.21.15.195:6827/2645959492","heartbeat_back_addr":"172.21.15.195:6831/2645959492","heartbeat_front_addr":"172.21.15.195:6829/2645959492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:41:41.708160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:04.276543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:27.085042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:50.425946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:11.756752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:34.703358+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:58.203757+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:44:22.277112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.149:6801/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3152364638":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/4228297071":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/869680670":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/1769813371":"2024-02-28T07:40:00.188096+0000","172.21.15.149:6800/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3694450196":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/2435290248":"2024-02-28T07:40:00.188096+0000","172.21.15.149:6800/1021120908":"2024-02-28T07:40:00.188096+0000","172.21.15.149:6801/1021120908":"2024-02-28T07:40:00.188096+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T07:44:27.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:26 smithi195 ceph-mon[25709]: osdmap e56: 8 total, 8 up, 8 in 2024-02-27T07:44:27.793 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-27T07:42:29.818441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-27T07:44:27.793 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-27T07:44:28.011 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:44:28.134 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[29527]: osdmap e57: 8 total, 8 up, 8 in 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3915555366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[29527]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[29527]: osdmap e58: 8 total, 8 up, 8 in 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[24796]: osdmap e57: 8 total, 8 up, 8 in 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3915555366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[24796]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:28.135 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:27 smithi149 ceph-mon[24796]: osdmap e58: 8 total, 8 up, 8 in 2024-02-27T07:44:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:27 smithi195 ceph-mon[25709]: osdmap e57: 8 total, 8 up, 8 in 2024-02-27T07:44:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:27 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3915555366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:44:28.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:27 smithi195 ceph-mon[25709]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:28.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:27 smithi195 ceph-mon[25709]: osdmap e58: 8 total, 8 up, 8 in 2024-02-27T07:44:29.646 INFO:teuthology.orchestra.run.smithi149.stdout:pg_num: 1 2024-02-27T07:44:30.314 INFO:tasks.cephadm:Adding prometheus.a on smithi195 2024-02-27T07:44:30.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply prometheus '1;smithi195=a' 2024-02-27T07:44:30.867 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:30 smithi195 ceph-mon[25709]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:30.867 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:30 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1932615016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-27T07:44:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:30 smithi149 ceph-mon[29527]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:30 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1932615016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-27T07:44:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:30 smithi149 ceph-mon[24796]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:30 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1932615016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-27T07:44:32.330 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled prometheus update... 2024-02-27T07:44:32.883 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:32 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-27T07:44:32.907 DEBUG:teuthology.orchestra.run.smithi195:prometheus.a> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service 2024-02-27T07:44:32.909 INFO:tasks.cephadm:Adding node-exporter.a on smithi149 2024-02-27T07:44:32.909 INFO:tasks.cephadm:Adding node-exporter.b on smithi195 2024-02-27T07:44:32.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply node-exporter '2;smithi149=a;smithi195=b' 2024-02-27T07:44:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-27T07:44:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' 2024-02-27T07:44:32.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:32 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-27T07:44:33.480 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ignoring --setuser ceph since I am not root 2024-02-27T07:44:33.480 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ignoring --setgroup ceph since I am not root 2024-02-27T07:44:33.480 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-mgr[26899]: -- 172.21.15.195:0/1542615961 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56197d270340 con 0x56197d249000 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[29527]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[29527]: Saving service prometheus spec with placement smithi195=a;count:1 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[29527]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[29527]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[24796]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[24796]: Saving service prometheus spec with placement smithi195=a;count:1 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[24796]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-27T07:44:33.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mon[24796]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-27T07:44:33.679 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setuser ceph since I am not root 2024-02-27T07:44:33.679 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setgroup ceph since I am not root 2024-02-27T07:44:33.679 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-mgr[25033]: -- 172.21.15.149:0/1808943537 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5641071ea340 con 0x5641071c2c00 2024-02-27T07:44:33.927 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-mon[25709]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:33.928 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-mon[25709]: Saving service prometheus spec with placement smithi195=a;count:1 2024-02-27T07:44:33.928 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-mon[25709]: from='mgr.14150 172.21.15.149:0/2819546050' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-27T07:44:33.928 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-mon[25709]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-27T07:44:34.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:33.885+0000 7faa3db86000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:44:34.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:33.975+0000 7faa3db86000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:44:34.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:33.925+0000 7fb0ac709000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:44:34.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:34.018+0000 7fb0ac709000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:44:34.817 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:34.634+0000 7fb0ac709000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:44:34.817 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:34.725+0000 7fb0ac709000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:44:34.817 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:34.814+0000 7fb0ac709000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:44:34.890 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:34.565+0000 7faa3db86000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:44:34.890 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:34.653+0000 7faa3db86000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:44:34.890 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:34.736+0000 7faa3db86000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:44:34.890 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:34.887+0000 7faa3db86000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:44:35.156 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:34.964+0000 7fb0ac709000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:44:35.156 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:35.046+0000 7fb0ac709000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:44:35.157 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:35.154+0000 7fb0ac709000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:44:35.168 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:34.968+0000 7faa3db86000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:44:35.168 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:35.082+0000 7faa3db86000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:44:35.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:35.166+0000 7faa3db86000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:44:35.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:35.331+0000 7faa3db86000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:44:35.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:35.415+0000 7faa3db86000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:44:35.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:35.237+0000 7fb0ac709000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:44:35.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:35.399+0000 7fb0ac709000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:44:35.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:35.483+0000 7fb0ac709000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:44:36.767 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:36.445+0000 7faa3db86000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:44:36.767 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:36.574+0000 7faa3db86000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:44:36.767 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:36.764+0000 7faa3db86000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:44:36.820 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:36.500+0000 7fb0ac709000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:44:36.820 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:36.629+0000 7fb0ac709000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:44:36.820 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:36.818+0000 7fb0ac709000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:44:37.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:36.852+0000 7faa3db86000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:44:37.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:36.988+0000 7faa3db86000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:44:37.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:36.904+0000 7fb0ac709000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:44:37.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:37.039+0000 7fb0ac709000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:44:37.622 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:37.312+0000 7faa3db86000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:44:37.622 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:37.533+0000 7faa3db86000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:44:37.661 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:37.357+0000 7fb0ac709000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:44:37.661 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:37.575+0000 7fb0ac709000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:44:37.661 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:37.659+0000 7fb0ac709000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:44:37.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:37.619+0000 7faa3db86000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:44:37.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:37.790+0000 7faa3db86000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:44:37.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:37.827+0000 7fb0ac709000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:44:39.009 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:38 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:38.710+0000 7faa3db86000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:44:39.023 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:38.731+0000 7fb0ac709000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:44:39.281 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:39.021+0000 7fb0ac709000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:44:39.282 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:39.115+0000 7fb0ac709000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:44:39.282 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:39.263+0000 7fb0ac709000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:44:39.282 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:39.333 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:39.007+0000 7faa3db86000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:44:39.333 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:39.105+0000 7faa3db86000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:44:39.333 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:39.256+0000 7faa3db86000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:44:39.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: Active manager daemon y restarted 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: Activating manager daemon y 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: osdmap e59: 8 total, 8 up, 8 in 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: Standby manager daemon x restarted 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: Active manager daemon y restarted 2024-02-27T07:44:39.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: Activating manager daemon y 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: osdmap e59: 8 total, 8 up, 8 in 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: Standby manager daemon x restarted 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: Standby manager daemon x started 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:44:39.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:44:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: Active manager daemon y restarted 2024-02-27T07:44:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: Activating manager daemon y 2024-02-27T07:44:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: osdmap e59: 8 total, 8 up, 8 in 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: Standby manager daemon x restarted 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/1933864818' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: CherryPy Checker: 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: The Application mounted at '' has an empty config. 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: [27/Feb/2024:07:44:39] ENGINE Serving on http://:::9283 2024-02-27T07:44:39.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:44:39 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: CherryPy Checker: 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: The Application mounted at '' has an empty config. 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Serving on http://:::9283 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:44:40.178 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:44:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:40.360 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled node-exporter update... 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: mgrmap e17: y(active, starting, since 0.0584506s), standbys: x 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:44:40.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: Manager daemon y is now available 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:40.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: mgrmap e17: y(active, starting, since 0.0584506s), standbys: x 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:44:40.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: Manager daemon y is now available 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:40.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: mgrmap e17: y(active, starting, since 0.0584506s), standbys: x 2024-02-27T07:44:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:44:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:44:40.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: Manager daemon y is now available 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:44:40.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:41.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: [27/Feb/2024:07:44:39] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: mgrmap e18: y(active, since 1.07051s), standbys: x 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi149=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: Saving service node-exporter spec with placement smithi149=a;smithi195=b;count:2 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.476 DEBUG:teuthology.orchestra.run.smithi149:node-exporter.a> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service 2024-02-27T07:44:41.478 DEBUG:teuthology.orchestra.run.smithi195:node-exporter.b> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service 2024-02-27T07:44:41.481 INFO:tasks.cephadm:Adding alertmanager.a on smithi149 2024-02-27T07:44:41.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply alertmanager '1;smithi149=a' 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: [27/Feb/2024:07:44:39] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: mgrmap e18: y(active, since 1.07051s), standbys: x 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi149=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: Saving service node-exporter spec with placement smithi149=a;smithi195=b;count:2 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: [27/Feb/2024:07:44:39] ENGINE Bus STARTING 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: [27/Feb/2024:07:44:39] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: [27/Feb/2024:07:44:39] ENGINE Bus STARTED 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: mgrmap e18: y(active, since 1.07051s), standbys: x 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi149=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: Saving service node-exporter spec with placement smithi149=a;smithi195=b;count:2 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:41.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:42 smithi195 ceph-mon[25709]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:42.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:42 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:42 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.373 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:42 smithi195 ceph-mon[25709]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-27T07:44:42.373 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:42 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[29527]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[29527]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[24796]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[24796]: mgrmap e19: y(active, since 2s), standbys: x 2024-02-27T07:44:42.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:42 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:43 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.617 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled alertmanager update... 2024-02-27T07:44:43.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:44:43.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:43.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:43 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:44.317 DEBUG:teuthology.orchestra.run.smithi149:alertmanager.a> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@alertmanager.a.service 2024-02-27T07:44:44.320 INFO:tasks.cephadm:Adding grafana.a on smithi195 2024-02-27T07:44:44.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph orch apply grafana '1;smithi195=a' 2024-02-27T07:44:44.599 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:44 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:44.599 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:44 smithi195 ceph-mon[25709]: Deploying daemon prometheus.a on smithi195 2024-02-27T07:44:44.599 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:44 smithi195 ceph-mon[25709]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:44.599 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:44 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:44.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[29527]: Deploying daemon prometheus.a on smithi195 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[29527]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[24796]: Deploying daemon prometheus.a on smithi195 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[24796]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:44.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:44 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:45 smithi149 ceph-mon[24796]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:45 smithi149 ceph-mon[24796]: Saving service alertmanager spec with placement smithi149=a;count:1 2024-02-27T07:44:45.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:45 smithi149 ceph-mon[29527]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:45.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:45 smithi149 ceph-mon[29527]: Saving service alertmanager spec with placement smithi149=a;count:1 2024-02-27T07:44:45.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:45 smithi195 ceph-mon[25709]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:45.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:45 smithi195 ceph-mon[25709]: Saving service alertmanager spec with placement smithi149=a;count:1 2024-02-27T07:44:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:46 smithi149 ceph-mon[24796]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:46 smithi149 ceph-mon[29527]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:46.695 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:46 smithi195 ceph-mon[25709]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:46.993 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled grafana update... 2024-02-27T07:44:47.991 DEBUG:teuthology.orchestra.run.smithi195:grafana.a> sudo journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@grafana.a.service 2024-02-27T07:44:47.993 INFO:tasks.cephadm:Setting up client nodes... 2024-02-27T07:44:47.995 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-27T07:44:48.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:47 smithi195 ceph-mon[25709]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:48.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:47 smithi195 ceph-mon[25709]: Saving service grafana spec with placement smithi195=a;count:1 2024-02-27T07:44:48.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:48.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:47 smithi195 ceph-mon[25709]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:48.293 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[29527]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:48.293 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[29527]: Saving service grafana spec with placement smithi195=a;count:1 2024-02-27T07:44:48.293 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:48.294 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[29527]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:48.294 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[24796]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:44:48.294 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[24796]: Saving service grafana spec with placement smithi195=a;count:1 2024-02-27T07:44:48.294 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:48.294 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:47 smithi149 ceph-mon[24796]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:50.023 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2024-02-27T07:44:50.023 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDykt1llU4UARAAZ/7Re34FPTVvMU/kRniDpQ== 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[24796]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1830552313' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[29527]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1830552313' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:50 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:50.671 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-27T07:44:50.671 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-27T07:44:50.671 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-27T07:44:50.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-27T07:44:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:50 smithi195 ceph-mon[25709]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:50 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1830552313' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:50 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:50 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:52.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:52 smithi195 ceph-mon[25709]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:52 smithi149 ceph-mon[29527]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:52.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:52 smithi149 ceph-mon[24796]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:53.870 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:53 smithi149 ceph-mon[29527]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:53.871 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:53 smithi149 ceph-mon[24796]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:53.885 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:53 smithi195 ceph-mon[25709]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:55.850 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2024-02-27T07:44:55.850 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD3kt1lG5k6MhAA7nAw30jZ95/RNVbkXeYcJg== 2024-02-27T07:44:56.391 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:56 smithi195 systemd[1]: Starting Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:44:56.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[29527]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:56.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[29527]: from='client.? 172.21.15.195:0/2864761302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[24796]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[24796]: from='client.? 172.21.15.195:0/2864761302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:56 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:56 smithi195 ceph-mon[25709]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:56 smithi195 ceph-mon[25709]: from='client.? 172.21.15.195:0/2864761302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:56 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T07:44:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:56 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T07:44:57.853 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-27T07:44:57.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-27T07:44:57.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-27T07:44:57.903 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-27T07:44:57.903 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-27T07:44:57.903 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph mgr dump --format=json 2024-02-27T07:44:58.075 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 podman[40812]: 2024-02-27 07:44:57.971862126 +0000 UTC m=+0.018895341 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-27T07:44:58.075 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 podman[40812]: 2024-02-27 07:44:58.073969621 +0000 UTC m=+0.121002817 container create 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:44:58.125 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:44:58.336 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 podman[40812]: 2024-02-27 07:44:58.309589582 +0000 UTC m=+0.356622781 container init 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:44:58.336 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 podman[40812]: 2024-02-27 07:44:58.312114127 +0000 UTC m=+0.359147332 container start 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:44:58.521 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:58 smithi149 ceph-mon[29527]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:58.522 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:58 smithi149 ceph-mon[24796]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:58.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-mon[25709]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:58.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi195 (none))" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.335Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.336Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.337Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.338Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.340Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.340Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.714µs 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.340Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-27T07:44:58.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.340Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.340Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.167µs wal_replay_duration=124.629µs total_replay_duration=157.591µs 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.341Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.341Z caller=main.go:947 level=info msg="TSDB started" 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.341Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.357Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.705983ms db_storage=689ns remote_storage=1.623µs web_handler=320ns query_engine=600ns scrape=358.952µs scrape_sd=16.557µs notify=810ns notify_sd=1.265µs rules=14.979782ms 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:44:58.357Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 bash[40812]: 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 2024-02-27T07:44:58.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:44:58 smithi195 systemd[1]: Started Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:44:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[24796]: Deploying daemon node-exporter.a on smithi149 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[24796]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[29527]: Deploying daemon node-exporter.a on smithi149 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[29527]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:59.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:44:59 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:59 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:59 smithi195 ceph-mon[25709]: Deploying daemon node-exporter.a on smithi149 2024-02-27T07:44:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:59 smithi195 ceph-mon[25709]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:44:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:44:59 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:44:59.927 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:44:59 smithi149 systemd[1]: Starting Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:45:00.018 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:00.303 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:44:59 smithi149 bash[44298]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-27T07:45:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:00 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/274524876' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-27T07:45:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:00 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/274524876' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-27T07:45:00.678 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:00 smithi149 bash[44298]: Getting image source signatures 2024-02-27T07:45:00.678 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:00 smithi149 bash[44298]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-27T07:45:00.678 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:00 smithi149 bash[44298]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-27T07:45:00.678 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:00 smithi149 bash[44298]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-27T07:45:00.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:00 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/274524876' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-27T07:45:00.772 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":19,"active_gid":14433,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":784752624},{"type":"v1","addr":"172.21.15.149:6801","nonce":784752624}]},"active_addr":"172.21.15.149:6801/784752624","active_change":"2024-02-27T07:44:39.270464+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34337,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.149:8443/","prometheus":"http://172.21.15.149:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":2945015925}]},{"addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":936127808}]},{"addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":2305001970}]},{"addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":1184246026}]}]}} 2024-02-27T07:45:00.775 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-27T07:45:00.775 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-27T07:45:00.776 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T07:45:01.124 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:01.597 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:01 smithi149 ceph-mon[24796]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:01.597 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:01 smithi149 ceph-mon[29527]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:01.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:01 smithi195 ceph-mon[25709]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:02.399 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:02 smithi149 bash[44298]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-27T07:45:02.677 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:02 smithi149 bash[44298]: Writing manifest to image destination 2024-02-27T07:45:03.093 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:02 smithi149 podman[44298]: 2024-02-27 07:45:02.746516878 +0000 UTC m=+2.865365446 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-27T07:45:03.093 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:02 smithi149 podman[44298]: 2024-02-27 07:45:02.986419415 +0000 UTC m=+3.105267980 container create fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:03.428 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 podman[44298]: 2024-02-27 07:45:03.172310858 +0000 UTC m=+3.291159424 container init fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:03.428 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 podman[44298]: 2024-02-27 07:45:03.175088254 +0000 UTC m=+3.293936823 container start fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.178Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.179Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.179Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.179Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=arp 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-27T07:45:03.429 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=edac 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-27T07:45:03.430 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=os 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=stat 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-27T07:45:03.431 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=time 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=uname 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[44458]: ts=2024-02-27T07:45:03.180Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 bash[44298]: fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 2024-02-27T07:45:03.432 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:45:03 smithi149 systemd[1]: Started Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:45:03.876 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:03.877 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":59,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","created":"2024-02-27T07:39:28.765884+0000","modified":"2024-02-27T07:44:39.270108+0000","last_up_change":"2024-02-27T07:44:23.774911+0000","last_in_change":"2024-02-27T07:44:07.023608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T07:42:29.818441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1fb57335-e582-4e40-82c3-5468f57f1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6803","nonce":2126977258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6805","nonce":2126977258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6809","nonce":2126977258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6807","nonce":2126977258}]},"public_addr":"172.21.15.149:6803/2126977258","cluster_addr":"172.21.15.149:6805/2126977258","heartbeat_back_addr":"172.21.15.149:6809/2126977258","heartbeat_front_addr":"172.21.15.149:6807/2126977258","state":["exists","up"]},{"osd":1,"uuid":"0a8e4a89-1b35-4c2d-856c-21450b2d55e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6811","nonce":3352774184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6813","nonce":3352774184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6817","nonce":3352774184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6815","nonce":3352774184}]},"public_addr":"172.21.15.149:6811/3352774184","cluster_addr":"172.21.15.149:6813/3352774184","heartbeat_back_addr":"172.21.15.149:6817/3352774184","heartbeat_front_addr":"172.21.15.149:6815/3352774184","state":["exists","up"]},{"osd":2,"uuid":"9b11afcf-4865-46e6-a6ae-9c96ab8580ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6819","nonce":4134394750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6821","nonce":4134394750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6825","nonce":4134394750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6823","nonce":4134394750}]},"public_addr":"172.21.15.149:6819/4134394750","cluster_addr":"172.21.15.149:6821/4134394750","heartbeat_back_addr":"172.21.15.149:6825/4134394750","heartbeat_front_addr":"172.21.15.149:6823/4134394750","state":["exists","up"]},{"osd":3,"uuid":"89097d49-cef0-48d1-a00a-e303f6cc9e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6827","nonce":3598442260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6829","nonce":3598442260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6833","nonce":3598442260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6831","nonce":3598442260}]},"public_addr":"172.21.15.149:6827/3598442260","cluster_addr":"172.21.15.149:6829/3598442260","heartbeat_back_addr":"172.21.15.149:6833/3598442260","heartbeat_front_addr":"172.21.15.149:6831/3598442260","state":["exists","up"]},{"osd":4,"uuid":"e3668d9a-3e64-4913-8fd4-563c9c996e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6801","nonce":1574773467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6803","nonce":1574773467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6807","nonce":1574773467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6805","nonce":1574773467}]},"public_addr":"172.21.15.195:6801/1574773467","cluster_addr":"172.21.15.195:6803/1574773467","heartbeat_back_addr":"172.21.15.195:6807/1574773467","heartbeat_front_addr":"172.21.15.195:6805/1574773467","state":["exists","up"]},{"osd":5,"uuid":"5f6b4d99-1f81-49ff-823e-535465189f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6809","nonce":4162967279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6811","nonce":4162967279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6815","nonce":4162967279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6813","nonce":4162967279}]},"public_addr":"172.21.15.195:6809/4162967279","cluster_addr":"172.21.15.195:6811/4162967279","heartbeat_back_addr":"172.21.15.195:6815/4162967279","heartbeat_front_addr":"172.21.15.195:6813/4162967279","state":["exists","up"]},{"osd":6,"uuid":"27bf5822-e0f1-495c-b54e-6a111eef979c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6817","nonce":822039124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6819","nonce":822039124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6823","nonce":822039124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6821","nonce":822039124}]},"public_addr":"172.21.15.195:6817/822039124","cluster_addr":"172.21.15.195:6819/822039124","heartbeat_back_addr":"172.21.15.195:6823/822039124","heartbeat_front_addr":"172.21.15.195:6821/822039124","state":["exists","up"]},{"osd":7,"uuid":"6af7cb6c-45bf-4b4b-804a-4fee7e774e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6825","nonce":2645959492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6827","nonce":2645959492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6831","nonce":2645959492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6829","nonce":2645959492}]},"public_addr":"172.21.15.195:6825/2645959492","cluster_addr":"172.21.15.195:6827/2645959492","heartbeat_back_addr":"172.21.15.195:6831/2645959492","heartbeat_front_addr":"172.21.15.195:6829/2645959492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:41:41.708160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:04.276543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:27.085042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:50.425946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:11.756752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:34.703358+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:58.203757+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:44:22.277112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.149:0/2661971708":"2024-02-28T07:44:39.270064+0000","172.21.15.149:0/2669522616":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/3373386519":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6801/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3152364638":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/4228297071":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/869680670":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/1769813371":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/1224151720":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3694450196":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/2435290248":"2024-02-28T07:40:00.188096+0000","172.21.15.149:6801/3373386519":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/1021120908":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/3043609035":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6801/1021120908":"2024-02-28T07:40:00.188096+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T07:45:04.326 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:04 smithi195 bash[41058]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-27T07:45:04.538 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-27T07:45:04.538 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T07:45:04.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:04 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:04.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:04 smithi195 ceph-mon[25709]: Deploying daemon node-exporter.b on smithi195 2024-02-27T07:45:04.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:04 smithi195 ceph-mon[25709]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:04.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:04 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2304564045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[29527]: Deploying daemon node-exporter.b on smithi195 2024-02-27T07:45:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[29527]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2304564045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:04.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:04.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[24796]: Deploying daemon node-exporter.b on smithi195 2024-02-27T07:45:04.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[24796]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:04.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:04 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2304564045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:04.758 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:04.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:04 smithi195 bash[41058]: Getting image source signatures 2024-02-27T07:45:04.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:04 smithi195 bash[41058]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-27T07:45:04.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:04 smithi195 bash[41058]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-27T07:45:04.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:04 smithi195 bash[41058]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-27T07:45:06.392 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 bash[41058]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-27T07:45:06.392 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 bash[41058]: Writing manifest to image destination 2024-02-27T07:45:06.654 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-mon[25709]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:06.655 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 podman[41058]: 2024-02-27 07:45:06.394517013 +0000 UTC m=+2.269489841 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-27T07:45:06.655 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 podman[41058]: 2024-02-27 07:45:06.485000339 +0000 UTC m=+2.359973152 container create 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:45:06.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:06 smithi149 ceph-mon[24796]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:06.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:06 smithi149 ceph-mon[29527]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:06.824 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:06.825 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":59,"fsid":"448a3a16-d543-11ee-95c3-87774f69a715","created":"2024-02-27T07:39:28.765884+0000","modified":"2024-02-27T07:44:39.270108+0000","last_up_change":"2024-02-27T07:44:23.774911+0000","last_in_change":"2024-02-27T07:44:07.023608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T07:42:29.818441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1fb57335-e582-4e40-82c3-5468f57f1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6803","nonce":2126977258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6805","nonce":2126977258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6809","nonce":2126977258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":2126977258},{"type":"v1","addr":"172.21.15.149:6807","nonce":2126977258}]},"public_addr":"172.21.15.149:6803/2126977258","cluster_addr":"172.21.15.149:6805/2126977258","heartbeat_back_addr":"172.21.15.149:6809/2126977258","heartbeat_front_addr":"172.21.15.149:6807/2126977258","state":["exists","up"]},{"osd":1,"uuid":"0a8e4a89-1b35-4c2d-856c-21450b2d55e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6811","nonce":3352774184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6813","nonce":3352774184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6817","nonce":3352774184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3352774184},{"type":"v1","addr":"172.21.15.149:6815","nonce":3352774184}]},"public_addr":"172.21.15.149:6811/3352774184","cluster_addr":"172.21.15.149:6813/3352774184","heartbeat_back_addr":"172.21.15.149:6817/3352774184","heartbeat_front_addr":"172.21.15.149:6815/3352774184","state":["exists","up"]},{"osd":2,"uuid":"9b11afcf-4865-46e6-a6ae-9c96ab8580ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6819","nonce":4134394750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6821","nonce":4134394750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6825","nonce":4134394750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":4134394750},{"type":"v1","addr":"172.21.15.149:6823","nonce":4134394750}]},"public_addr":"172.21.15.149:6819/4134394750","cluster_addr":"172.21.15.149:6821/4134394750","heartbeat_back_addr":"172.21.15.149:6825/4134394750","heartbeat_front_addr":"172.21.15.149:6823/4134394750","state":["exists","up"]},{"osd":3,"uuid":"89097d49-cef0-48d1-a00a-e303f6cc9e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6827","nonce":3598442260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6829","nonce":3598442260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6833","nonce":3598442260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":3598442260},{"type":"v1","addr":"172.21.15.149:6831","nonce":3598442260}]},"public_addr":"172.21.15.149:6827/3598442260","cluster_addr":"172.21.15.149:6829/3598442260","heartbeat_back_addr":"172.21.15.149:6833/3598442260","heartbeat_front_addr":"172.21.15.149:6831/3598442260","state":["exists","up"]},{"osd":4,"uuid":"e3668d9a-3e64-4913-8fd4-563c9c996e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6801","nonce":1574773467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6803","nonce":1574773467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6807","nonce":1574773467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":1574773467},{"type":"v1","addr":"172.21.15.195:6805","nonce":1574773467}]},"public_addr":"172.21.15.195:6801/1574773467","cluster_addr":"172.21.15.195:6803/1574773467","heartbeat_back_addr":"172.21.15.195:6807/1574773467","heartbeat_front_addr":"172.21.15.195:6805/1574773467","state":["exists","up"]},{"osd":5,"uuid":"5f6b4d99-1f81-49ff-823e-535465189f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6809","nonce":4162967279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6811","nonce":4162967279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6815","nonce":4162967279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":4162967279},{"type":"v1","addr":"172.21.15.195:6813","nonce":4162967279}]},"public_addr":"172.21.15.195:6809/4162967279","cluster_addr":"172.21.15.195:6811/4162967279","heartbeat_back_addr":"172.21.15.195:6815/4162967279","heartbeat_front_addr":"172.21.15.195:6813/4162967279","state":["exists","up"]},{"osd":6,"uuid":"27bf5822-e0f1-495c-b54e-6a111eef979c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6817","nonce":822039124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6819","nonce":822039124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6823","nonce":822039124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":822039124},{"type":"v1","addr":"172.21.15.195:6821","nonce":822039124}]},"public_addr":"172.21.15.195:6817/822039124","cluster_addr":"172.21.15.195:6819/822039124","heartbeat_back_addr":"172.21.15.195:6823/822039124","heartbeat_front_addr":"172.21.15.195:6821/822039124","state":["exists","up"]},{"osd":7,"uuid":"6af7cb6c-45bf-4b4b-804a-4fee7e774e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6825","nonce":2645959492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6827","nonce":2645959492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6831","nonce":2645959492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2645959492},{"type":"v1","addr":"172.21.15.195:6829","nonce":2645959492}]},"public_addr":"172.21.15.195:6825/2645959492","cluster_addr":"172.21.15.195:6827/2645959492","heartbeat_back_addr":"172.21.15.195:6831/2645959492","heartbeat_front_addr":"172.21.15.195:6829/2645959492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:41:41.708160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:04.276543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:27.085042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:42:50.425946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:11.756752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:34.703358+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:43:58.203757+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T07:44:22.277112+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.149:0/2661971708":"2024-02-28T07:44:39.270064+0000","172.21.15.149:0/2669522616":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/3373386519":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6801/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3152364638":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/4228297071":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/869680670":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/1769813371":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/1224151720":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/3583506788":"2024-02-28T07:40:22.682158+0000","172.21.15.149:0/3694450196":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/2435290248":"2024-02-28T07:40:00.188096+0000","172.21.15.149:6801/3373386519":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6800/1021120908":"2024-02-28T07:40:00.188096+0000","172.21.15.149:0/3043609035":"2024-02-28T07:44:39.270064+0000","172.21.15.149:6801/1021120908":"2024-02-28T07:40:00.188096+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T07:45:06.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:06.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 podman[41058]: 2024-02-27 07:45:06.65358704 +0000 UTC m=+2.528559863 container init 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:45:06.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 podman[41058]: 2024-02-27 07:45:06.656161393 +0000 UTC m=+2.531134213 container start 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:45:06.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.659Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-27T07:45:06.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.660Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-27T07:45:06.965 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.660Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.660Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=arp 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-27T07:45:06.966 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=edac 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-27T07:45:06.967 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=os 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=stat 2024-02-27T07:45:06.968 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=time 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=uname 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[41111]: ts=2024-02-27T07:45:06.661Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 bash[41058]: 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 2024-02-27T07:45:06.969 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:45:06 smithi195 systemd[1]: Started Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:45:07.510 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-27T07:45:07.510 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-27T07:45:07.511 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-27T07:45:07.511 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-27T07:45:07.511 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-27T07:45:07.512 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-27T07:45:07.512 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-27T07:45:07.513 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2076360465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[24796]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:07.830 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:07] "GET /metrics HTTP/1.1" 200 191515 "" "Prometheus/2.33.4" 2024-02-27T07:45:07.970 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.971 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.971 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.972 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.987 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.990 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:07.998 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:08.006 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:08.125 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:08.126 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:45:08.126 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:45:08.126 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:45:08.126 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2076360465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:08.126 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:07 smithi149 ceph-mon[29527]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2076360465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T07:45:08.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:07 smithi195 ceph-mon[25709]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:10.653 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[29527]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:10.654 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:10.654 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:10.654 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[24796]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:10.654 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:10.654 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:10 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:10.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:10 smithi195 ceph-mon[25709]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:10.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:10 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:10.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:10 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:12.674 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:12 smithi149 ceph-mon[24796]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:12.675 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:12 smithi149 ceph-mon[29527]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:12 smithi195 ceph-mon[25709]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:13.815 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:13 smithi149 ceph-mon[24796]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:13.815 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:13 smithi149 ceph-mon[29527]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:13.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:13 smithi195 ceph-mon[25709]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:16.671 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:16 smithi195 ceph-mon[25709]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:16.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:16 smithi149 ceph-mon[29527]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:16.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:16 smithi149 ceph-mon[24796]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:16.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:17.867 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:17] "GET /metrics HTTP/1.1" 200 191516 "" "Prometheus/2.33.4" 2024-02-27T07:45:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:18 smithi149 ceph-mon[24796]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:18.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:18 smithi149 ceph-mon[29527]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:18.621 INFO:teuthology.orchestra.run.smithi149.stdout:231928233997 2024-02-27T07:45:18.622 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-27T07:45:18.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:18 smithi195 ceph-mon[25709]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:20.430 INFO:teuthology.orchestra.run.smithi149.stdout:107374182431 2024-02-27T07:45:20.430 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-27T07:45:20.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:20 smithi149 ceph-mon[24796]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:20.662 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:20 smithi149 ceph-mon[29527]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:20.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:20 smithi195 ceph-mon[25709]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:21.087 INFO:teuthology.orchestra.run.smithi149.stdout:34359738412 2024-02-27T07:45:21.087 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-27T07:45:22.141 INFO:teuthology.orchestra.run.smithi149.stdout:55834574889 2024-02-27T07:45:22.142 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-27T07:45:22.342 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:22.664 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:22 smithi149 ceph-mon[29527]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:22.664 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:22 smithi149 ceph-mon[24796]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:22.691 INFO:teuthology.orchestra.run.smithi149.stdout:197568495633 2024-02-27T07:45:22.691 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-27T07:45:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:22 smithi195 ceph-mon[25709]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:23 smithi149 ceph-mon[29527]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:23.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:23 smithi149 ceph-mon[24796]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:23.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:23 smithi195 ceph-mon[25709]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:24.852 INFO:teuthology.orchestra.run.smithi149.stdout:77309411365 2024-02-27T07:45:24.852 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-27T07:45:25.800 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:25.801 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:25.802 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:26.188 INFO:teuthology.orchestra.run.smithi149.stdout:163208757272 2024-02-27T07:45:26.189 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-27T07:45:26.209 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:26.210 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:26.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[29527]: Deploying daemon alertmanager.a on smithi149 2024-02-27T07:45:26.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[29527]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[24796]: Deploying daemon alertmanager.a on smithi149 2024-02-27T07:45:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:26 smithi149 ceph-mon[24796]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:26.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:26.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:26 smithi195 ceph-mon[25709]: Deploying daemon alertmanager.a on smithi149 2024-02-27T07:45:26.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:26 smithi195 ceph-mon[25709]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:26.620 INFO:teuthology.orchestra.run.smithi149.stdout:128849018908 2024-02-27T07:45:26.621 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-27T07:45:26.641 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:26.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:27.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:27] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-02-27T07:45:28.651 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:28 smithi149 ceph-mon[24796]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:28.651 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:28 smithi149 ceph-mon[29527]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:28.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:28 smithi195 ceph-mon[25709]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:29.091 INFO:teuthology.orchestra.run.smithi149.stdout:231928233999 2024-02-27T07:45:29.577 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:29 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2888748710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-27T07:45:29.578 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:29 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2888748710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-27T07:45:29.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:29 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2888748710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-27T07:45:30.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:30 smithi149 ceph-mon[24796]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:30.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:30 smithi149 ceph-mon[29527]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:30.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:30 smithi195 ceph-mon[25709]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:31.735 INFO:teuthology.orchestra.run.smithi149.stdout:77309411367 2024-02-27T07:45:32.510 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411367 for osd.2 2024-02-27T07:45:32.510 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:32.615 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:32.641 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928233999 for osd.7 2024-02-27T07:45:32.641 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:32.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:32 smithi149 ceph-mon[24796]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:32.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:32 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1284884843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-27T07:45:32.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:32 smithi149 ceph-mon[29527]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:32.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:32 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1284884843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-27T07:45:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:32 smithi195 ceph-mon[25709]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:32 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1284884843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-27T07:45:33.861 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:33 smithi195 ceph-mon[25709]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:33.879 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:33 smithi149 ceph-mon[24796]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:33.880 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:33 smithi149 ceph-mon[29527]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:35.299 INFO:teuthology.orchestra.run.smithi149.stdout:34359738415 2024-02-27T07:45:35.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:35 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1849407024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-27T07:45:35.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:35 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1849407024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-27T07:45:35.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:35 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1849407024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-27T07:45:36.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:36 smithi149 ceph-mon[29527]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:36.547 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:36 smithi149 ceph-mon[24796]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:36.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:36 smithi195 ceph-mon[25709]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:36.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:37.753 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:37] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-02-27T07:45:37.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738415 for osd.0 2024-02-27T07:45:37.779 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:38.574 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:38 smithi149 ceph-mon[24796]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:38.574 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:38 smithi149 ceph-mon[29527]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:38.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:38 smithi195 ceph-mon[25709]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:39.063 INFO:teuthology.orchestra.run.smithi149.stdout:197568495637 2024-02-27T07:45:39.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:39 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3749088584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-27T07:45:39.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:39 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3749088584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-27T07:45:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:39 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3749088584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-27T07:45:39.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495633 got 197568495637 for osd.6 2024-02-27T07:45:39.942 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[24796]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[29527]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:40.661 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:40 smithi195 ceph-mon[25709]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:45:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:45:41.713 INFO:teuthology.orchestra.run.smithi149.stdout:107374182436 2024-02-27T07:45:42.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:42 smithi149 ceph-mon[29527]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:42.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:42 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/671087312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-27T07:45:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:42 smithi149 ceph-mon[24796]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:42.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:42 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/671087312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-27T07:45:42.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:42 smithi195 ceph-mon[25709]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:42.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:42 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/671087312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-27T07:45:43.460 INFO:teuthology.orchestra.run.smithi149.stdout:163208757276 2024-02-27T07:45:43.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:43 smithi149 ceph-mon[29527]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:43.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:43 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3896791203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-27T07:45:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:43 smithi149 ceph-mon[24796]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:43.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:43 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3896791203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-27T07:45:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:43 smithi195 ceph-mon[25709]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:43.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:43 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3896791203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-27T07:45:44.199 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757276 for osd.5 2024-02-27T07:45:44.199 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:44.639 INFO:teuthology.orchestra.run.smithi149.stdout:128849018912 2024-02-27T07:45:44.943 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:44 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1997472401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-27T07:45:44.944 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:44 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1997472401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-27T07:45:44.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182431 got 107374182436 for osd.3 2024-02-27T07:45:44.945 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:44.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:44 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1997472401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-27T07:45:45.509 INFO:teuthology.orchestra.run.smithi149.stdout:55834574894 2024-02-27T07:45:45.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:45 smithi149 ceph-mon[29527]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:45.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:45 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2577906035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-27T07:45:45.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:45 smithi149 ceph-mon[24796]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:45.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:45 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2577906035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-27T07:45:45.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:45 smithi195 ceph-mon[25709]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:45.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:45 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2577906035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-27T07:45:46.119 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574894 for osd.1 2024-02-27T07:45:46.120 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:46.588 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018912 for osd.4 2024-02-27T07:45:46.588 DEBUG:teuthology.parallel:result is None 2024-02-27T07:45:46.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-27T07:45:46.589 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-27T07:45:46.874 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:46 smithi149 podman[47502]: 2024-02-27 07:45:46.796288244 +0000 UTC m=+0.195212976 volume create 6cc8d0e489a157fa8c8a4a4c6b4db783067805fbc7f1a8d4d728d0c1ce0e6aa5 2024-02-27T07:45:46.874 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:46 smithi149 podman[47502]: 2024-02-27 07:45:46.873492904 +0000 UTC m=+0.272417662 container create 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:46.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:47.350 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:47.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 podman[47502]: 2024-02-27 07:45:47.159609995 +0000 UTC m=+0.558534710 container init 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 podman[47502]: 2024-02-27 07:45:47.162265693 +0000 UTC m=+0.561190419 container start 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.173Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.173Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.174Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.149 port=9094 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.176Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.200Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.200Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.203Z caller=main.go:518 msg=Listening address=:9093 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:47.203Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 bash[47502]: 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 2024-02-27T07:45:47.428 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:47 smithi149 systemd[1]: Started Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:45:47.748 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:47] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-02-27T07:45:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[29527]: Deploying daemon grafana.a on smithi195 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:48 smithi149 ceph-mon[24796]: Deploying daemon grafana.a on smithi195 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:48.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:48 smithi195 ceph-mon[25709]: Deploying daemon grafana.a on smithi195 2024-02-27T07:45:48.945 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:48.946 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2024-02-27T07:45:49.523 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:49 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:49.177Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001029806s 2024-02-27T07:45:49.547 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-02-27T07:45:47.346344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48824,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912392,"statfs":{"total":767960285184,"available":767910289408,"internally_reserved":0,"allocated":6217728,"data_stored":3486852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002197"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-27T07:44:39.431073+0000","last_change":"2024-02-27T07:44:27.801934+0000","last_active":"2024-02-27T07:44:39.431073+0000","last_peered":"2024-02-27T07:44:39.431073+0000","last_clean":"2024-02-27T07:44:39.431073+0000","last_became_active":"2024-02-27T07:44:27.801467+0000","last_became_peered":"2024-02-27T07:44:27.801467+0000","last_unstale":"2024-02-27T07:44:39.431073+0000","last_undegraded":"2024-02-27T07:44:39.431073+0000","last_fullsized":"2024-02-27T07:44:39.431073+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-27T07:42:29.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-27T07:42:29.993428+0000","last_clean_scrub_stamp":"2024-02-27T07:42:29.993428+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-28T15:33:31.410417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.439},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.754}]},{"osd":2,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.68500000000000005}]},{"osd":3,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.72799999999999998}]},{"osd":4,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.56299999999999994}]},{"osd":5,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.70799999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":8.4079999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":8.3960000000000008}]},{"osd":1,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.498}]},{"osd":2,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.41599999999999998}]},{"osd":3,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.55000000000000004}]},{"osd":7,"last update":"Tue Feb 27 07:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.52100000000000002,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.5},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.39300000000000002,"15min":0.379},"min":{"1min":0.24399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.57999999999999996,"5min":0.625,"15min":0.625},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Feb 27 07:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59199999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Feb 27 07:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.315,"5min":0.312,"15min":0.312},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Feb 27 07:45:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.73999999999999999}]},{"osd":5,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.50900000000000001,"5min":0.496,"15min":0.496},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67300000000000004}]},{"osd":6,"last update":"Tue Feb 27 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.70699999999999996}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 27 07:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33000000000000002,"15min":0.315},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35099999999999998,"15min":0.32500000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.30499999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.379,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.504,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.45600000000000002}]},{"osd":3,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.434}]},{"osd":4,"last update":"Tue Feb 27 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66600000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79000000000000004,"5min":0.878,"15min":0.878},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79200000000000004,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.61899999999999999}]},{"osd":6,"last update":"Tue Feb 27 07:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68100000000000005}]},{"osd":7,"last update":"Tue Feb 27 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":9.6530000000000005,"5min":9.6530000000000005,"15min":9.6530000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.72899999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38700000000000001,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.56100000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58499999999999996,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.374}]},{"osd":1,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.41099999999999998,"15min":0.42599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54700000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Feb 27 07:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64200000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.435},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47099999999999997,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40899999999999997}]},{"osd":4,"last update":"Tue Feb 27 07:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79300000000000004,"5min":0.872,"15min":0.872},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.62,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57999999999999996}]},{"osd":5,"last update":"Tue Feb 27 07:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.80100000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63}]},{"osd":6,"last update":"Tue Feb 27 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57299999999999995}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.66800000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39800000000000002,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.503}]},{"osd":1,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.37},{"interface":"front","average":{"1min":0.438,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59599999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.745},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57599999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.42499999999999999}]},{"osd":4,"last update":"Tue Feb 27 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78100000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.71599999999999997}]},{"osd":5,"last update":"Tue Feb 27 07:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.495,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80400000000000005,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.63},{"interface":"front","average":{"1min":0.627,"5min":0.621,"15min":0.62},"min":{"1min":0.43099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86699999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86699999999999999}]},{"osd":6,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Tue Feb 27 07:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.66100000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64900000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59799999999999998}]},{"osd":1,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66900000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.504}]},{"osd":3,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65500000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.60699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58799999999999997}]},{"osd":5,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.29599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.64500000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.437},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Feb 27 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.62}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.39600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.751},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.59599999999999997,"15min":0.54300000000000004},"min":{"1min":0.38700000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":6.1539999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"last":0.625}]},{"osd":1,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.60299999999999998,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1710000000000003,"5min":6.1710000000000003,"15min":6.1710000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.85399999999999998,"15min":0.90200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85099999999999998,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":0.66900000000000004}]},{"osd":2,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.622,"15min":0.57599999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":6.1269999999999998,"5min":6.1269999999999998,"15min":6.1269999999999998},"last":0.73999999999999999}]},{"osd":3,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.41599999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.47899999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.71099999999999997}]},{"osd":4,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.435,"15min":0.42099999999999999},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69299999999999995}]},{"osd":6,"last update":"Tue Feb 27 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.64100000000000001}]},{"osd":7,"last update":"Tue Feb 27 07:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.78200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-27T07:45:49.549 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-27T07:45:49.765 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:50.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[29527]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:50.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[29527]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:50.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:50.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[24796]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:50.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[24796]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:50.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:50 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:50 smithi195 ceph-mon[25709]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:50 smithi195 ceph-mon[25709]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:50 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:45:51.468 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:51.471 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2024-02-27T07:45:52.327 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-27T07:45:51.346978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48824,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912392,"statfs":{"total":767960285184,"available":767910289408,"internally_reserved":0,"allocated":6217728,"data_stored":3486852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002092"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-27T07:44:39.431073+0000","last_change":"2024-02-27T07:44:27.801934+0000","last_active":"2024-02-27T07:44:39.431073+0000","last_peered":"2024-02-27T07:44:39.431073+0000","last_clean":"2024-02-27T07:44:39.431073+0000","last_became_active":"2024-02-27T07:44:27.801467+0000","last_became_peered":"2024-02-27T07:44:27.801467+0000","last_unstale":"2024-02-27T07:44:39.431073+0000","last_undegraded":"2024-02-27T07:44:39.431073+0000","last_fullsized":"2024-02-27T07:44:39.431073+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-27T07:42:29.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-27T07:42:29.993428+0000","last_clean_scrub_stamp":"2024-02-27T07:42:29.993428+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-28T15:33:31.410417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.56499999999999995}]},{"osd":1,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52600000000000002}]},{"osd":2,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.503},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.55200000000000005}]},{"osd":3,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.58199999999999996}]},{"osd":4,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.32900000000000001}]},{"osd":5,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40899999999999997}]},{"osd":6,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.373},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.31}]},{"osd":3,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.439}]},{"osd":4,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.32500000000000001}]},{"osd":5,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42999999999999999}]},{"osd":7,"last update":"Tue Feb 27 07:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.501}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.52100000000000002,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.379},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.39300000000000002,"15min":0.379},"min":{"1min":0.24399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.57999999999999996,"5min":0.625,"15min":0.625},"last":0.30599999999999999}]},{"osd":2,"last update":"Tue Feb 27 07:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59199999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40100000000000002}]},{"osd":3,"last update":"Tue Feb 27 07:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.315,"5min":0.312,"15min":0.312},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Feb 27 07:45:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.49299999999999999}]},{"osd":5,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.50900000000000001,"5min":0.496,"15min":0.496},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue Feb 27 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55400000000000005}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 27 07:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33000000000000002,"15min":0.315},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35099999999999998,"15min":0.32500000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.247}]},{"osd":2,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.30499999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.379,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.504,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.41199999999999998}]},{"osd":3,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.35199999999999998}]},{"osd":4,"last update":"Tue Feb 27 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66600000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79000000000000004,"5min":0.878,"15min":0.878},"last":0.38700000000000001}]},{"osd":5,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79200000000000004,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.373},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.40100000000000002}]},{"osd":6,"last update":"Tue Feb 27 07:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.436}]},{"osd":7,"last update":"Tue Feb 27 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":9.6530000000000005,"5min":9.6530000000000005,"15min":9.6530000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38700000000000001,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.56100000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58499999999999996,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.372}]},{"osd":1,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.41099999999999998,"15min":0.42599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54700000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Tue Feb 27 07:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64200000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47099999999999997,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40000000000000002}]},{"osd":4,"last update":"Tue Feb 27 07:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79300000000000004,"5min":0.872,"15min":0.872},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.62,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.47599999999999998}]},{"osd":5,"last update":"Tue Feb 27 07:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.433},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.80100000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54100000000000004}]},{"osd":6,"last update":"Tue Feb 27 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49299999999999999}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39800000000000002,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34999999999999998}]},{"osd":1,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.312},{"interface":"front","average":{"1min":0.438,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Tue Feb 27 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59599999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57599999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.33900000000000002}]},{"osd":4,"last update":"Tue Feb 27 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78100000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.432}]},{"osd":5,"last update":"Tue Feb 27 07:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.495,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80400000000000005,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.621,"15min":0.62},"min":{"1min":0.43099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86699999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.39000000000000001}]},{"osd":6,"last update":"Tue Feb 27 07:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.497},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.40500000000000003}]},{"osd":7,"last update":"Tue Feb 27 07:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46899999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64900000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.504}]},{"osd":1,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66900000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52500000000000002}]},{"osd":3,"last update":"Tue Feb 27 07:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65500000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.60699999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Tue Feb 27 07:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.29599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.64500000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.432}]},{"osd":6,"last update":"Tue Feb 27 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.41699999999999998}]},{"osd":7,"last update":"Tue Feb 27 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.371},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40799999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.39600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.59599999999999997,"15min":0.54300000000000004},"min":{"1min":0.38700000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":6.1539999999999999,"5min":6.1539999999999999,"15min":6.1539999999999999},"last":0.38}]},{"osd":1,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.60299999999999998,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1710000000000003,"5min":6.1710000000000003,"15min":6.1710000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.85399999999999998,"15min":0.90200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85099999999999998,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.622,"15min":0.57599999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":6.1269999999999998,"5min":6.1269999999999998,"15min":6.1269999999999998},"last":0.33900000000000002}]},{"osd":3,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.41599999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.47899999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Feb 27 07:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.435,"15min":0.42099999999999999},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42899999999999999}]},{"osd":6,"last update":"Tue Feb 27 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.46500000000000002}]},{"osd":7,"last update":"Tue Feb 27 07:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.495},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-27T07:45:52.328 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-27T07:45:52.328 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-27T07:45:52.328 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-27T07:45:52.329 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- ceph health --format=json 2024-02-27T07:45:52.553 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/mon.a/config 2024-02-27T07:45:52.581 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:52 smithi149 ceph-mon[24796]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:52.581 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:52 smithi149 ceph-mon[29527]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:52.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:52 smithi195 ceph-mon[25709]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:53.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:53 smithi149 ceph-mon[29527]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:53.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:53 smithi149 ceph-mon[24796]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:53.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:53 smithi195 ceph-mon[25709]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T07:45:54.312 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-27T07:45:54.312 INFO:teuthology.orchestra.run.smithi149.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-27T07:45:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:54 smithi149 ceph-mon[29527]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:54 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1053404271' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-27T07:45:54.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:54 smithi149 ceph-mon[24796]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:54 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1053404271' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-27T07:45:54.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:54 smithi195 ceph-mon[25709]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:54 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1053404271' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-27T07:45:54.927 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-27T07:45:54.927 INFO:tasks.cephadm:Setup complete, yielding 2024-02-27T07:45:54.927 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T07:45:54.939 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'lock': False, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'verbose': True, 'machine_type': 'smithi', 'os_type': None, 'owner': 'scheduled_adking@teuthology', 'interactive_on_error': False, 'suite_path': None, 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'os_version': None, 'block': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574539', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'mon.a': ['ceph config set mgr mgr/cephadm/use_repo_digest false --force']}}, {'cephadm.shell': {'env': ['sha1'], 'mon.a': ['radosgw-admin realm create --rgw-realm=r --default', 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default', 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default', 'radosgw-admin period update --rgw-realm=r --commit', 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000', 'ceph orch apply rgw smpl', 'ceph osd pool create foo', 'rbd pool init foo', 'ceph orch apply iscsi foo u p', 'sleep 120', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'mon.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph versions', 'ceph orch upgrade status', 'ceph health detail', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1", "ceph orch ls | grep '^osd '"]}}, {'cephadm.shell': {'mon.a': ['ceph orch upgrade ls', 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0', 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']], [Remote(name='ubuntu@smithi195.front.sepia.ceph.com'), ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCskd1lovrDBBAA0xYDOJpP6SvAF/xPbbtVfw==\n', bootstrap_remote=Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '448a3a16-d543-11ee-95c3-87774f69a715', 'mon election default strategy': 1}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': 'true', 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 448a3a16-d543-11ee-95c3-87774f69a715\n[global]\n\tfsid = 448a3a16-d543-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]\n', first_mgr='y', first_mon='a', first_mon_role='mon.a', fsid='448a3a16-d543-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCrkd1ls4yDDxAAI5L1lY3CbYkl+aQJJduxzA==\n\tcaps mon = "allow *"\n', mons={'mon.a': '172.21.15.149', 'mon.c': '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'mon.b': '172.21.15.195'}, roleless=False, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'mon.a': ['ceph config set mgr mgr/cephadm/use_repo_digest false --force']}, 'cluster_name': ''} 2024-02-27T07:45:54.939 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T07:45:54.939 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-02-27T07:45:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:56 smithi195 ceph-mon[25709]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:56.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:45:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:45:56.804 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:56 smithi149 ceph-mon[24796]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:56.804 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:56 smithi149 ceph-mon[29527]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:57.349 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:45:57.178Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002889959s 2024-02-27T07:45:57.374 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T07:45:57.383 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'lock': False, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'verbose': True, 'machine_type': 'smithi', 'os_type': None, 'owner': 'scheduled_adking@teuthology', 'interactive_on_error': False, 'suite_path': None, 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'os_version': None, 'block': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574539', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'mon.a': ['ceph config set mgr mgr/cephadm/use_repo_digest false --force']}}, {'cephadm.shell': {'mon.a': ['radosgw-admin realm create --rgw-realm=r --default', 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default', 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default', 'radosgw-admin period update --rgw-realm=r --commit', 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000', 'ceph orch apply rgw smpl', 'ceph osd pool create foo', 'rbd pool init foo', 'ceph orch apply iscsi foo u p', 'sleep 120', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'mon.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph versions', 'ceph orch upgrade status', 'ceph health detail', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1", "ceph orch ls | grep '^osd '"]}}, {'cephadm.shell': {'mon.a': ['ceph orch upgrade ls', 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0', 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']], [Remote(name='ubuntu@smithi195.front.sepia.ceph.com'), ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCskd1lovrDBBAA0xYDOJpP6SvAF/xPbbtVfw==\n', bootstrap_remote=Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '448a3a16-d543-11ee-95c3-87774f69a715', 'mon election default strategy': 1}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': 'true', 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 448a3a16-d543-11ee-95c3-87774f69a715\n[global]\n\tfsid = 448a3a16-d543-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]\n', first_mgr='y', first_mon='a', first_mon_role='mon.a', fsid='448a3a16-d543-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCrkd1ls4yDDxAAI5L1lY3CbYkl+aQJJduxzA==\n\tcaps mon = "allow *"\n', mons={'mon.a': '172.21.15.149', 'mon.c': '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'mon.b': '172.21.15.195'}, roleless=False, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'mon.a': ['radosgw-admin realm create --rgw-realm=r --default', 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default', 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default', 'radosgw-admin period update --rgw-realm=r --commit', 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000', 'ceph orch apply rgw smpl', 'ceph osd pool create foo', 'rbd pool init foo', 'ceph orch apply iscsi foo u p', 'sleep 120', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}, 'cluster_name': ''} 2024-02-27T07:45:57.383 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T07:45:57.384 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-27T07:45:57.879 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' 2024-02-27T07:45:57.879 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-mon[29527]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:57.879 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:45:57] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-02-27T07:45:57.879 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' 2024-02-27T07:45:57.880 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:45:57 smithi149 ceph-mon[24796]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:57.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:57 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' 2024-02-27T07:45:57.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:45:57 smithi195 ceph-mon[25709]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:45:59.854 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-27T07:46:00.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-27T07:46:00.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-27T07:46:00.216 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-27T07:46:00.217 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-27T07:46:00.217 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-27T07:46:00.217 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-27T07:46:00.217 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-27T07:46:00.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-27T07:46:00.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-27T07:46:00.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-27T07:46:00.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-27T07:46:00.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-27T07:46:00.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-27T07:46:00.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-27T07:46:00.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-27T07:46:00.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-27T07:46:00.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=175.232805ms 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:45:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:45:59+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:46:00+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:46:00+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:46:00+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:46:00+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-27T07:46:00.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:46:00+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b24efc94-e7fb-4c72-8b4e-b81632043abf", 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "r", 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: "current_period": "fefd6f55-960a-4c42-bd23-13649a176f59", 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 1 2024-02-27T07:46:00.361 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:46:00.593 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: osdmap e60: 8 total, 8 up, 8 in 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2974796617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:00.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:00 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: osdmap e60: 8 total, 8 up, 8 in 2024-02-27T07:46:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2974796617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: osdmap e60: 8 total, 8 up, 8 in 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2974796617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:00 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:01 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-27T07:46:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:01 smithi149 ceph-mon[29527]: osdmap e61: 8 total, 8 up, 8 in 2024-02-27T07:46:01.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:01 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-27T07:46:01.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:01 smithi149 ceph-mon[24796]: osdmap e61: 8 total, 8 up, 8 in 2024-02-27T07:46:01.473 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-27T07:46:01.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:01 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-27T07:46:01.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:01 smithi195 ceph-mon[25709]: osdmap e61: 8 total, 8 up, 8 in 2024-02-27T07:46:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:02 smithi149 ceph-mon[29527]: osdmap e62: 8 total, 8 up, 8 in 2024-02-27T07:46:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:02 smithi149 ceph-mon[29527]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:02 smithi149 ceph-mon[24796]: osdmap e62: 8 total, 8 up, 8 in 2024-02-27T07:46:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:02 smithi149 ceph-mon[24796]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:02 smithi195 ceph-mon[25709]: osdmap e62: 8 total, 8 up, 8 in 2024-02-27T07:46:02.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:02 smithi195 ceph-mon[25709]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "5d8199f5-c37a-4cab-9ef7-62586379d37d", 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default", 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "default", 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": "true", 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [], 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "", 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [], 2024-02-27T07:46:03.169 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [], 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "", 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "b24efc94-e7fb-4c72-8b4e-b81632043abf", 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:03.170 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:46:04.350 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:04 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:04.350 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:04 smithi149 ceph-mon[24796]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:04.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:04 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:04.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:04 smithi149 ceph-mon[29527]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:04.373 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-27T07:46:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:04 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:04.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:04 smithi195 ceph-mon[25709]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-27T07:46:05.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:05 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:05 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:05 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:05.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:05 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:06.463 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:46:06.463 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "c08d8a5b-5fa0-4497-acdf-8aeb031c1191", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "z", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "z.rgw.meta:root", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "z.rgw.control", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "z.rgw.log", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-27T07:46:06.464 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "z.rgw.otp", 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "", 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "" 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:06.465 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "b24efc94-e7fb-4c72-8b4e-b81632043abf", 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "z.rgw.log:notif" 2024-02-27T07:46:06.466 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:46:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[24796]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 679 B/s wr, 1 op/s 2024-02-27T07:46:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[24796]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:46:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[24796]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:46:06.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[29527]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 679 B/s wr, 1 op/s 2024-02-27T07:46:06.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[29527]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:46:06.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:06 smithi149 ceph-mon[29527]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:46:06.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:46:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:46:06.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:06 smithi195 ceph-mon[25709]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 679 B/s wr, 1 op/s 2024-02-27T07:46:06.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:06 smithi195 ceph-mon[25709]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:46:06.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:06 smithi195 ceph-mon[25709]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:46:07.031 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-27T07:46:07.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:46:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:07] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-02-27T07:46:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:08 smithi149 ceph-mon[24796]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:08 smithi149 ceph-mon[29527]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:08.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:08 smithi195 ceph-mon[25709]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:10 smithi149 ceph-mon[29527]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-27T07:46:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:10 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:10 smithi149 ceph-mon[24796]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-27T07:46:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:10 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:10 smithi195 ceph-mon[25709]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-27T07:46:10.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:10 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:11.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 systemd[1]: Stopping Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:46:11.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 bash[49336]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager.a" found: no such container 2024-02-27T07:46:11.712 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[47542]: level=info ts=2024-02-27T07:46:11.461Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-27T07:46:11.713 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 podman[49347]: 2024-02-27 07:46:11.481405928 +0000 UTC m=+0.149511001 container died 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:12.130 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 podman[49347]: 2024-02-27 07:46:11.869593717 +0000 UTC m=+0.537698786 container cleanup 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:12.130 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 bash[49347]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a 2024-02-27T07:46:12.130 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:11 smithi149 bash[49374]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager.a" found: no such container 2024-02-27T07:46:12.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49362]: 2024-02-27 07:46:12.129154732 +0000 UTC m=+0.647409343 container remove 820751e92affa6a848f5aa6b016cb39ccdd0e293de819ee2db35b393fcba87c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:12.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49362]: 2024-02-27 07:46:12.204616694 +0000 UTC m=+0.722871271 volume remove 6cc8d0e489a157fa8c8a4a4c6b4db783067805fbc7f1a8d4d728d0c1ce0e6aa5 2024-02-27T07:46:12.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-27T07:46:12.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 systemd[1]: Stopped Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:46:12.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 systemd[1]: Starting Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:46:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:12 smithi195 ceph-mon[25709]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 715 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:12 smithi195 ceph-mon[25709]: osdmap e63: 8 total, 8 up, 8 in 2024-02-27T07:46:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:12 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:12 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.737 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[24796]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 715 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:12.737 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[24796]: osdmap e63: 8 total, 8 up, 8 in 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[29527]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 715 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[29527]: osdmap e63: 8 total, 8 up, 8 in 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.738 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-27T07:46:12.738 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49442]: 2024-02-27 07:46:12.336878987 +0000 UTC m=+0.017891704 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-27T07:46:12.738 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49442]: 2024-02-27 07:46:12.437894857 +0000 UTC m=+0.118907567 volume create 6ba6d3e1a02a54d8fe6c69fe3a9ebf44b0d3649aada7b0d73e5f819098da27da 2024-02-27T07:46:12.738 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49442]: 2024-02-27 07:46:12.555798897 +0000 UTC m=+0.236811602 container create 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:13.177 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49442]: 2024-02-27 07:46:12.833209627 +0000 UTC m=+0.514222334 container init 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:13.177 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 podman[49442]: 2024-02-27 07:46:12.8359096 +0000 UTC m=+0.516922318 container start 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:13.177 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.845Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-27T07:46:13.177 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.845Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.846Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.149 port=9094 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.848Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.872Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.872Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.875Z caller=main.go:518 msg=Listening address=:9093 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:12.875Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 bash[49442]: 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 2024-02-27T07:46:13.178 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:12 smithi149 systemd[1]: Started Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:46:13.655 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:13 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-27T07:46:13.655 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:13 smithi195 ceph-mon[25709]: osdmap e64: 8 total, 8 up, 8 in 2024-02-27T07:46:13.655 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:13 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:13.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-27T07:46:13.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[29527]: osdmap e64: 8 total, 8 up, 8 in 2024-02-27T07:46:13.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:13.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-27T07:46:13.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[24796]: osdmap e64: 8 total, 8 up, 8 in 2024-02-27T07:46:13.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:13 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: osdmap e65: 8 total, 8 up, 8 in 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-27T07:46:14.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:14 smithi195 ceph-mon[25709]: osdmap e66: 8 total, 8 up, 8 in 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: osdmap e65: 8 total, 8 up, 8 in 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-27T07:46:14.850 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[29527]: osdmap e66: 8 total, 8 up, 8 in 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: osdmap e65: 8 total, 8 up, 8 in 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2740382496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-27T07:46:14.851 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-mon[24796]: osdmap e66: 8 total, 8 up, 8 in 2024-02-27T07:46:15.177 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:14.849Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000931765s 2024-02-27T07:46:15.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:15 smithi195 ceph-mon[25709]: pgmap v58: 97 pgs: 38 unknown, 59 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:15.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:15 smithi149 ceph-mon[29527]: pgmap v58: 97 pgs: 38 unknown, 59 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:15.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:15 smithi149 ceph-mon[24796]: pgmap v58: 97 pgs: 38 unknown, 59 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:16.143 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:15 smithi195 systemd[1]: Stopping Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:46:16.143 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 bash[42489]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-27T07:46:16.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[40827]: ts=2024-02-27T07:46:16.142Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-27T07:46:16.466 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 podman[42502]: 2024-02-27 07:46:16.152276455 +0000 UTC m=+0.118424018 container died 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:16.735 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 podman[42502]: 2024-02-27 07:46:16.483528142 +0000 UTC m=+0.449675717 container cleanup 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:16.735 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 bash[42502]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a 2024-02-27T07:46:16.736 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 podman[42516]: 2024-02-27 07:46:16.701515246 +0000 UTC m=+0.550975474 container remove 9763f9b626860fd772f482cf02a0ba6aa1128d10ae314e2dcd687fe198457389 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:16.736 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-mon[25709]: osdmap e67: 8 total, 8 up, 8 in 2024-02-27T07:46:16.736 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:16.736 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:16 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:16.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[29527]: osdmap e67: 8 total, 8 up, 8 in 2024-02-27T07:46:16.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:16.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[24796]: osdmap e67: 8 total, 8 up, 8 in 2024-02-27T07:46:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:16.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:16 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 bash[42528]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 systemd[1]: Stopped Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 systemd[1]: Starting Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:16 smithi195 podman[42596]: 2024-02-27 07:46:16.870156174 +0000 UTC m=+0.017906102 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-27T07:46:17.003 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 podman[42596]: 2024-02-27 07:46:17.002422659 +0000 UTC m=+0.150172586 container create 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 podman[42596]: 2024-02-27 07:46:17.221097427 +0000 UTC m=+0.368847359 container init 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 podman[42596]: 2024-02-27 07:46:17.223675179 +0000 UTC m=+0.371425113 container start 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi195 (none))" 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.247Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.249Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-27T07:46:17.254 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.249Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-27T07:46:17.255 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.250Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-27T07:46:17.255 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.252Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-27T07:46:17.255 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.252Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.152µs 2024-02-27T07:46:17.255 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.252Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 bash[42596]: 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 systemd[1]: Started Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.456Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.456Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.456Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.759µs wal_replay_duration=203.344751ms total_replay_duration=203.402831ms 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.458Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.458Z caller=main.go:947 level=info msg="TSDB started" 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.458Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.476Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.504278ms db_storage=1.033µs remote_storage=1.365µs web_handler=384ns query_engine=588ns scrape=388.191µs scrape_sd=22.282µs notify=24.709µs notify_sd=7.821µs rules=17.80584ms 2024-02-27T07:46:17.516 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:46:17.476Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: osdmap e68: 8 total, 8 up, 8 in 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.517 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:17.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:17 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:17.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: osdmap e68: 8 total, 8 up, 8 in 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:17.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: osdmap e68: 8 total, 8 up, 8 in 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.149:9093"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.195:9095"}]: dispatch 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.195:3000"}]: dispatch 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:17.931 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:17 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "2e94d916-d6b4-4e15-a6ab-49a4e93a501c", 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 1, 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "predecessor_uuid": "fefd6f55-960a-4c42-bd23-13649a176f59", 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": [], 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "period_map": { 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "2e94d916-d6b4-4e15-a6ab-49a4e93a501c", 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "zonegroups": [ 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "5d8199f5-c37a-4cab-9ef7-62586379d37d", 2024-02-27T07:46:18.537 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "default", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": "true", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [], 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "c08d8a5b-5fa0-4497-acdf-8aeb031c1191", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [ 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "c08d8a5b-5fa0-4497-acdf-8aeb031c1191", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "z", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [], 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": "false", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": "false", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": "false", 2024-02-27T07:46:18.538 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": "true", 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "" 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [ 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD" 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:46:18.539 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "b24efc94-e7fb-4c72-8b4e-b81632043abf", 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "short_zone_ids": [ 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "c08d8a5b-5fa0-4497-acdf-8aeb031c1191", 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 496131482 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zonegroup": "5d8199f5-c37a-4cab-9ef7-62586379d37d", 2024-02-27T07:46:18.540 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "c08d8a5b-5fa0-4497-acdf-8aeb031c1191", 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "period_config": { 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_quota": { 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "user_quota": { 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-02-27T07:46:18.541 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "user_ratelimit": { 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_ratelimit": { 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.542 INFO:teuthology.orchestra.run.smithi149.stdout: "anonymous_ratelimit": { 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "b24efc94-e7fb-4c72-8b4e-b81632043abf", 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_name": "r", 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-02-27T07:46:18.543 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:46:18.848 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-27T07:46:18.848 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[24796]: osdmap e69: 8 total, 8 up, 8 in 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[29527]: osdmap e69: 8 total, 8 up, 8 in 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:18.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:18 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:18.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:18 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-27T07:46:18.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:18 smithi195 ceph-mon[25709]: osdmap e69: 8 total, 8 up, 8 in 2024-02-27T07:46:18.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:18 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3599321243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:18.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:18 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-27T07:46:19.257 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-02-27T07:46:19.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-27T07:46:19.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[24796]: osdmap e70: 8 total, 8 up, 8 in 2024-02-27T07:46:19.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[24796]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:19.762 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-27T07:46:19.762 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[29527]: osdmap e70: 8 total, 8 up, 8 in 2024-02-27T07:46:19.762 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:19 smithi149 ceph-mon[29527]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:19.959 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:19 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-27T07:46:19.960 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:19 smithi195 ceph-mon[25709]: osdmap e70: 8 total, 8 up, 8 in 2024-02-27T07:46:19.960 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:19 smithi195 ceph-mon[25709]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-27T07:46:22.008 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:21 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:22.008 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:21 smithi149 ceph-mon[24796]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-27T07:46:22.009 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:21 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:22.009 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:21 smithi149 ceph-mon[29527]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-27T07:46:22.130 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled rgw.foo update... 2024-02-27T07:46:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:21 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:22.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:21 smithi195 ceph-mon[25709]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-27T07:46:22.805 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch apply rgw smpl' 2024-02-27T07:46:23.129 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:46:22.851Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002986121s 2024-02-27T07:46:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:23.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[24796]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:23.430 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:23 smithi149 ceph-mon[29527]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: Saving service rgw.foo spec with placement count:2 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:23.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:23 smithi195 ceph-mon[25709]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:46:24.362 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:24 smithi149 ceph-mon[29527]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-27T07:46:24.362 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:24 smithi149 ceph-mon[24796]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-27T07:46:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:24 smithi195 ceph-mon[25709]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-27T07:46:26.455 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled rgw.smpl update... 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:26.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:26 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:26.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:46:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:46:27.158 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph osd pool create foo' 2024-02-27T07:46:27.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:27 smithi195 ceph-mon[25709]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:46:27.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:27 smithi195 ceph-mon[25709]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:27.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:27 smithi195 ceph-mon[25709]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:27.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:27 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:27.751 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:27] "GET /metrics HTTP/1.1" 200 196871 "" "Prometheus/2.33.4" 2024-02-27T07:46:27.752 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[24796]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:46:27.752 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[24796]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:27.752 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[24796]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:27.752 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:27.752 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[29527]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:46:27.753 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[29527]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:27.753 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[29527]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:27.753 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:27 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:28.569 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:28 smithi149 ceph-mon[24796]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 725 B/s rd, 725 B/s wr, 1 op/s 2024-02-27T07:46:28.569 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:28 smithi149 ceph-mon[29527]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 725 B/s rd, 725 B/s wr, 1 op/s 2024-02-27T07:46:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:28 smithi195 ceph-mon[25709]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 725 B/s rd, 725 B/s wr, 1 op/s 2024-02-27T07:46:29.476 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'foo' created 2024-02-27T07:46:29.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:29 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/784187172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:29 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:29 smithi195 ceph-mon[25709]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 659 B/s wr, 1 op/s 2024-02-27T07:46:29.770 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/784187172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.770 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.770 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[29527]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 659 B/s wr, 1 op/s 2024-02-27T07:46:29.770 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/784187172' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.771 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-27T07:46:29.771 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:29 smithi149 ceph-mon[24796]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 659 B/s wr, 1 op/s 2024-02-27T07:46:30.094 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'rbd pool init foo' 2024-02-27T07:46:30.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:30 smithi149 ceph-mon[29527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-27T07:46:30.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:30 smithi149 ceph-mon[29527]: osdmap e71: 8 total, 8 up, 8 in 2024-02-27T07:46:30.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:30 smithi149 ceph-mon[24796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-27T07:46:30.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:30 smithi149 ceph-mon[24796]: osdmap e71: 8 total, 8 up, 8 in 2024-02-27T07:46:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:30 smithi195 ceph-mon[25709]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-27T07:46:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:30 smithi195 ceph-mon[25709]: osdmap e71: 8 total, 8 up, 8 in 2024-02-27T07:46:31.554 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:31 smithi149 ceph-mon[29527]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T07:46:31.554 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:31 smithi149 ceph-mon[29527]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-02-27T07:46:31.554 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:31 smithi149 ceph-mon[24796]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T07:46:31.554 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:31 smithi149 ceph-mon[24796]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-02-27T07:46:31.694 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:31 smithi195 ceph-mon[25709]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T07:46:31.694 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:31 smithi195 ceph-mon[25709]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[24796]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[29527]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:32.621 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:32.622 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:32 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:32 smithi195 ceph-mon[25709]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T07:46:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:32 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-27T07:46:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:32 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:32 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:32.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:32 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:33.794 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:33 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-27T07:46:33.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:33 smithi195 ceph-mon[25709]: osdmap e74: 8 total, 8 up, 8 in 2024-02-27T07:46:33.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:33 smithi195 ceph-mon[25709]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-02-27T07:46:33.795 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-27T07:46:33.795 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[24796]: osdmap e74: 8 total, 8 up, 8 in 2024-02-27T07:46:33.795 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[24796]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-02-27T07:46:33.796 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/440622789' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-27T07:46:33.796 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[29527]: osdmap e74: 8 total, 8 up, 8 in 2024-02-27T07:46:33.796 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:33 smithi149 ceph-mon[29527]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-02-27T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:34 smithi195 ceph-mon[25709]: osdmap e75: 8 total, 8 up, 8 in 2024-02-27T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:34 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:34.801 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:34 smithi149 ceph-mon[29527]: osdmap e75: 8 total, 8 up, 8 in 2024-02-27T07:46:34.801 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:34 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:34.801 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:34 smithi149 ceph-mon[24796]: osdmap e75: 8 total, 8 up, 8 in 2024-02-27T07:46:34.801 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:34 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:35.537 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch apply iscsi foo u p' 2024-02-27T07:46:35.858 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:35 smithi149 ceph-mon[24796]: osdmap e76: 8 total, 8 up, 8 in 2024-02-27T07:46:35.858 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:35 smithi149 ceph-mon[24796]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-02-27T07:46:35.858 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:35 smithi149 ceph-mon[29527]: osdmap e76: 8 total, 8 up, 8 in 2024-02-27T07:46:35.858 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:35 smithi149 ceph-mon[29527]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-02-27T07:46:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:35 smithi195 ceph-mon[25709]: osdmap e76: 8 total, 8 up, 8 in 2024-02-27T07:46:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:35 smithi195 ceph-mon[25709]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-02-27T07:46:36.910 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.910 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.910 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:36.910 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[24796]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.911 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:36.912 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:36 smithi149 ceph-mon[29527]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:46:36.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: Saving service rgw.smpl spec with placement count:2 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:36.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:36.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:36.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:36 smithi195 ceph-mon[25709]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:46:37.545 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled iscsi.foo update... 2024-02-27T07:46:37.911 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:46:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:37] "GET /metrics HTTP/1.1" 200 196871 "" "Prometheus/2.33.4" 2024-02-27T07:46:37.911 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:37 smithi149 ceph-mon[29527]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 522 B/s wr, 170 op/s 2024-02-27T07:46:37.911 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:37 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:37.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:37 smithi149 ceph-mon[24796]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 522 B/s wr, 170 op/s 2024-02-27T07:46:37.911 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:37 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:37.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:37 smithi195 ceph-mon[25709]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 522 B/s wr, 170 op/s 2024-02-27T07:46:37.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:37 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:38.361 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'sleep 120' 2024-02-27T07:46:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:38 smithi195 ceph-mon[25709]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:38 smithi195 ceph-mon[25709]: Saving service iscsi.foo spec with placement count:1 2024-02-27T07:46:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:38 smithi195 ceph-mon[25709]: osdmap e77: 8 total, 8 up, 8 in 2024-02-27T07:46:38.997 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[24796]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:38.997 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[24796]: Saving service iscsi.foo spec with placement count:1 2024-02-27T07:46:38.997 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[24796]: osdmap e77: 8 total, 8 up, 8 in 2024-02-27T07:46:38.997 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[29527]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:46:38.997 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[29527]: Saving service iscsi.foo spec with placement count:1 2024-02-27T07:46:38.998 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:38 smithi149 ceph-mon[29527]: osdmap e77: 8 total, 8 up, 8 in 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:39.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:39.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:39.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:39.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:39.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:39.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:40.001 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:40.002 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:46:40.003 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:40.004 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:46:41.037 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-27T07:46:41.037 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-27T07:46:41.037 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-27T07:46:41.037 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-27T07:46:41.037 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[29527]: osdmap e78: 8 total, 8 up, 8 in 2024-02-27T07:46:41.038 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-27T07:46:41.038 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-27T07:46:41.038 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-27T07:46:41.038 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-27T07:46:41.038 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:40 smithi149 ceph-mon[24796]: osdmap e78: 8 total, 8 up, 8 in 2024-02-27T07:46:41.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-27T07:46:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-27T07:46:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-27T07:46:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-27T07:46:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:40 smithi195 ceph-mon[25709]: osdmap e78: 8 total, 8 up, 8 in 2024-02-27T07:46:41.910 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: osdmap e79: 8 total, 8 up, 8 in 2024-02-27T07:46:41.910 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:41.911 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:41 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:42.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: osdmap e79: 8 total, 8 up, 8 in 2024-02-27T07:46:42.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:42.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:42.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: osdmap e79: 8 total, 8 up, 8 in 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:42.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:41 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:43.055 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:42 smithi149 ceph-mon[29527]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:46:43.055 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:42 smithi149 ceph-mon[24796]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:46:43.155 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:42 smithi195 ceph-mon[25709]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:46:43.775 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:43 smithi195 ceph-mon[25709]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:43.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:43 smithi149 ceph-mon[29527]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:43.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:43 smithi149 ceph-mon[24796]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-27T07:46:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:45.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:44 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:44 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:44 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:45.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:44 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:45.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:44 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:45.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:45 smithi195 ceph-mon[25709]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 535 B/s wr, 268 op/s 2024-02-27T07:46:46.139 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:45 smithi149 ceph-mon[24796]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 535 B/s wr, 268 op/s 2024-02-27T07:46:46.140 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:45 smithi149 ceph-mon[29527]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 535 B/s wr, 268 op/s 2024-02-27T07:46:46.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:46:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:46:47.920 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:47] "GET /metrics HTTP/1.1" 200 208021 "" "Prometheus/2.33.4" 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.921 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-27T07:46:47.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:47 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:47.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-27T07:46:47.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:47 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:49.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[24796]: Checking pool "foo" exists for service iscsi.foo 2024-02-27T07:46:49.562 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[24796]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:46:49.562 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[24796]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-27T07:46:49.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[29527]: Checking pool "foo" exists for service iscsi.foo 2024-02-27T07:46:49.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[29527]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:46:49.562 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:49 smithi149 ceph-mon[29527]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-27T07:46:49.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:49 smithi195 ceph-mon[25709]: Checking pool "foo" exists for service iscsi.foo 2024-02-27T07:46:49.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:49 smithi195 ceph-mon[25709]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:46:49.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:49 smithi195 ceph-mon[25709]: mgrmap e20: y(active, since 2m), standbys: x 2024-02-27T07:46:50.336 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:50 smithi149 ceph-mon[29527]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 424 B/s wr, 213 op/s 2024-02-27T07:46:50.336 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:50 smithi149 ceph-mon[24796]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 424 B/s wr, 213 op/s 2024-02-27T07:46:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:50 smithi195 ceph-mon[25709]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 424 B/s wr, 213 op/s 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3586868045' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3619389768' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:51.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:51.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:51.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3586868045' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:46:51.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3619389768' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:51.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:51 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3586868045' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3619389768' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:51 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]: dispatch 2024-02-27T07:46:52.394 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[24796]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 192 B/s wr, 192 op/s 2024-02-27T07:46:52.395 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]': finished 2024-02-27T07:46:52.395 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[24796]: osdmap e80: 8 total, 8 up, 8 in 2024-02-27T07:46:52.395 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]: dispatch 2024-02-27T07:46:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[29527]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 192 B/s wr, 192 op/s 2024-02-27T07:46:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]': finished 2024-02-27T07:46:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[29527]: osdmap e80: 8 total, 8 up, 8 in 2024-02-27T07:46:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:52 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]: dispatch 2024-02-27T07:46:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:52 smithi195 ceph-mon[25709]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 192 B/s wr, 192 op/s 2024-02-27T07:46:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:52 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2661971708"}]': finished 2024-02-27T07:46:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:52 smithi195 ceph-mon[25709]: osdmap e80: 8 total, 8 up, 8 in 2024-02-27T07:46:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:52 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]: dispatch 2024-02-27T07:46:53.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:46:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:46:53.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:46:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]': finished 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[24796]: osdmap e81: 8 total, 8 up, 8 in 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]: dispatch 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]': finished 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[29527]: osdmap e81: 8 total, 8 up, 8 in 2024-02-27T07:46:53.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:53 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]: dispatch 2024-02-27T07:46:53.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:53 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3833289039' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2669522616"}]': finished 2024-02-27T07:46:53.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:53 smithi195 ceph-mon[25709]: osdmap e81: 8 total, 8 up, 8 in 2024-02-27T07:46:53.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:53 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]: dispatch 2024-02-27T07:46:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]': finished 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: osdmap e82: 8 total, 8 up, 8 in 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:54.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2339600029' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]': finished 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: osdmap e82: 8 total, 8 up, 8 in 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:54.681 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2339600029' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:54.682 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:54 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:54.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2276233605' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3373386519"}]': finished 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: osdmap e82: 8 total, 8 up, 8 in 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2339600029' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:54.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:54 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]: dispatch 2024-02-27T07:46:55.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: Adding iSCSI gateway http://:@172.21.15.149:5000 to Dashboard 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]': finished 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: osdmap e83: 8 total, 8 up, 8 in 2024-02-27T07:46:55.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:55 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]: dispatch 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: Adding iSCSI gateway http://:@172.21.15.149:5000 to Dashboard 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]': finished 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: osdmap e83: 8 total, 8 up, 8 in 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]: dispatch 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: Adding iSCSI gateway http://:@172.21.15.149:5000 to Dashboard 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi149"}]: dispatch 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: Checking dashboard <-> RGW credentials 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3583506788"}]': finished 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: osdmap e83: 8 total, 8 up, 8 in 2024-02-27T07:46:55.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:55 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]: dispatch 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-mon[25709]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]': finished 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-mon[25709]: osdmap e84: 8 total, 8 up, 8 in 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2970837393' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:56.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:56 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[24796]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]': finished 2024-02-27T07:46:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[24796]: osdmap e84: 8 total, 8 up, 8 in 2024-02-27T07:46:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2970837393' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:56.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[29527]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:56.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/229829591' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3152364638"}]': finished 2024-02-27T07:46:56.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[29527]: osdmap e84: 8 total, 8 up, 8 in 2024-02-27T07:46:56.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2970837393' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:56.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:56 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]: dispatch 2024-02-27T07:46:57.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:46:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:46:57] "GET /metrics HTTP/1.1" 200 214937 "" "Prometheus/2.33.4" 2024-02-27T07:46:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]': finished 2024-02-27T07:46:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[24796]: osdmap e85: 8 total, 8 up, 8 in 2024-02-27T07:46:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3483581456' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[24796]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]': finished 2024-02-27T07:46:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[29527]: osdmap e85: 8 total, 8 up, 8 in 2024-02-27T07:46:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3483581456' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:58 smithi149 ceph-mon[29527]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:58.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:58 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4228297071"}]': finished 2024-02-27T07:46:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:58 smithi195 ceph-mon[25709]: osdmap e85: 8 total, 8 up, 8 in 2024-02-27T07:46:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:58 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3483581456' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:58 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]: dispatch 2024-02-27T07:46:58.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:58 smithi195 ceph-mon[25709]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-27T07:46:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]': finished 2024-02-27T07:46:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[29527]: osdmap e86: 8 total, 8 up, 8 in 2024-02-27T07:46:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]: dispatch 2024-02-27T07:46:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]': finished 2024-02-27T07:46:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[24796]: osdmap e86: 8 total, 8 up, 8 in 2024-02-27T07:46:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:46:59 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]: dispatch 2024-02-27T07:46:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:59 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/869680670"}]': finished 2024-02-27T07:46:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:59 smithi195 ceph-mon[25709]: osdmap e86: 8 total, 8 up, 8 in 2024-02-27T07:46:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:46:59 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]: dispatch 2024-02-27T07:47:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]': finished 2024-02-27T07:47:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[29527]: osdmap e87: 8 total, 8 up, 8 in 2024-02-27T07:47:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]: dispatch 2024-02-27T07:47:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[29527]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:47:00.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]': finished 2024-02-27T07:47:00.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[24796]: osdmap e87: 8 total, 8 up, 8 in 2024-02-27T07:47:00.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]: dispatch 2024-02-27T07:47:00.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:00 smithi149 ceph-mon[24796]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:47:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:00 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1929988164' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1769813371"}]': finished 2024-02-27T07:47:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:00 smithi195 ceph-mon[25709]: osdmap e87: 8 total, 8 up, 8 in 2024-02-27T07:47:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:00 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]: dispatch 2024-02-27T07:47:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:00 smithi195 ceph-mon[25709]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:47:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]': finished 2024-02-27T07:47:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[29527]: osdmap e88: 8 total, 8 up, 8 in 2024-02-27T07:47:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]: dispatch 2024-02-27T07:47:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]': finished 2024-02-27T07:47:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[24796]: osdmap e88: 8 total, 8 up, 8 in 2024-02-27T07:47:01.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:01 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]: dispatch 2024-02-27T07:47:01.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:01 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/377650197' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1224151720"}]': finished 2024-02-27T07:47:01.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:01 smithi195 ceph-mon[25709]: osdmap e88: 8 total, 8 up, 8 in 2024-02-27T07:47:01.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:01 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]: dispatch 2024-02-27T07:47:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]': finished 2024-02-27T07:47:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[29527]: osdmap e89: 8 total, 8 up, 8 in 2024-02-27T07:47:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]: dispatch 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[29527]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]': finished 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[24796]: osdmap e89: 8 total, 8 up, 8 in 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]: dispatch 2024-02-27T07:47:02.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:02 smithi149 ceph-mon[24796]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:02 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:02 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2524079885' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/3583506788"}]': finished 2024-02-27T07:47:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:02 smithi195 ceph-mon[25709]: osdmap e89: 8 total, 8 up, 8 in 2024-02-27T07:47:02.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:02 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]: dispatch 2024-02-27T07:47:02.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:02 smithi195 ceph-mon[25709]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:03.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:03 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]': finished 2024-02-27T07:47:03.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:03 smithi195 ceph-mon[25709]: osdmap e90: 8 total, 8 up, 8 in 2024-02-27T07:47:03.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:03 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4167876927' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:03 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]': finished 2024-02-27T07:47:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[29527]: osdmap e90: 8 total, 8 up, 8 in 2024-02-27T07:47:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4167876927' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3968182536' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3694450196"}]': finished 2024-02-27T07:47:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[24796]: osdmap e90: 8 total, 8 up, 8 in 2024-02-27T07:47:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4167876927' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]: dispatch 2024-02-27T07:47:03.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:47:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:47:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:47:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]': finished 2024-02-27T07:47:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[29527]: osdmap e91: 8 total, 8 up, 8 in 2024-02-27T07:47:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[29527]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]: dispatch 2024-02-27T07:47:04.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]': finished 2024-02-27T07:47:04.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[24796]: osdmap e91: 8 total, 8 up, 8 in 2024-02-27T07:47:04.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[24796]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:04.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:04 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]: dispatch 2024-02-27T07:47:04.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:04 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2435290248"}]': finished 2024-02-27T07:47:04.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:04 smithi195 ceph-mon[25709]: osdmap e91: 8 total, 8 up, 8 in 2024-02-27T07:47:04.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:04 smithi195 ceph-mon[25709]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:04.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:04 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]: dispatch 2024-02-27T07:47:05.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]': finished 2024-02-27T07:47:05.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[29527]: osdmap e92: 8 total, 8 up, 8 in 2024-02-27T07:47:05.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]: dispatch 2024-02-27T07:47:05.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]': finished 2024-02-27T07:47:05.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[24796]: osdmap e92: 8 total, 8 up, 8 in 2024-02-27T07:47:05.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:05 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]: dispatch 2024-02-27T07:47:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:05 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1545052498' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/3373386519"}]': finished 2024-02-27T07:47:05.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:05 smithi195 ceph-mon[25709]: osdmap e92: 8 total, 8 up, 8 in 2024-02-27T07:47:05.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:05 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]: dispatch 2024-02-27T07:47:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]': finished 2024-02-27T07:47:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[29527]: osdmap e93: 8 total, 8 up, 8 in 2024-02-27T07:47:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[29527]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]: dispatch 2024-02-27T07:47:06.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]': finished 2024-02-27T07:47:06.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[24796]: osdmap e93: 8 total, 8 up, 8 in 2024-02-27T07:47:06.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[24796]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:06.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:06 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]: dispatch 2024-02-27T07:47:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:06 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2976985652' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/1021120908"}]': finished 2024-02-27T07:47:06.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:06 smithi195 ceph-mon[25709]: osdmap e93: 8 total, 8 up, 8 in 2024-02-27T07:47:06.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:06 smithi195 ceph-mon[25709]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:06.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:06 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]: dispatch 2024-02-27T07:47:06.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:07.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]': finished 2024-02-27T07:47:07.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[29527]: osdmap e94: 8 total, 8 up, 8 in 2024-02-27T07:47:07.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]: dispatch 2024-02-27T07:47:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]': finished 2024-02-27T07:47:07.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[24796]: osdmap e94: 8 total, 8 up, 8 in 2024-02-27T07:47:07.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]: dispatch 2024-02-27T07:47:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:07 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3910027460' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/3043609035"}]': finished 2024-02-27T07:47:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:07 smithi195 ceph-mon[25709]: osdmap e94: 8 total, 8 up, 8 in 2024-02-27T07:47:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:07 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]: dispatch 2024-02-27T07:47:07.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:07] "GET /metrics HTTP/1.1" 200 214937 "" "Prometheus/2.33.4" 2024-02-27T07:47:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]': finished 2024-02-27T07:47:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[29527]: osdmap e95: 8 total, 8 up, 8 in 2024-02-27T07:47:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[29527]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:08.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]': finished 2024-02-27T07:47:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[24796]: osdmap e95: 8 total, 8 up, 8 in 2024-02-27T07:47:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:08 smithi149 ceph-mon[24796]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:08 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2062325194' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/1021120908"}]': finished 2024-02-27T07:47:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:08 smithi195 ceph-mon[25709]: osdmap e95: 8 total, 8 up, 8 in 2024-02-27T07:47:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:08 smithi195 ceph-mon[25709]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:10 smithi149 ceph-mon[29527]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-27T07:47:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:10 smithi149 ceph-mon[24796]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-27T07:47:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:10 smithi195 ceph-mon[25709]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-27T07:47:12.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:12 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:12.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:12 smithi149 ceph-mon[29527]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:47:12.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:12 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:12.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:12 smithi149 ceph-mon[24796]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:47:12.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:12 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:12.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:12 smithi195 ceph-mon[25709]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:47:13.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:13 smithi195 ceph-mon[25709]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:47:14.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:13 smithi149 ceph-mon[29527]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:47:14.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:13 smithi149 ceph-mon[24796]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:47:16.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:16 smithi149 ceph-mon[29527]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:47:16.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:16 smithi149 ceph-mon[24796]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:47:16.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:16.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:16 smithi195 ceph-mon[25709]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:47:17.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:17] "GET /metrics HTTP/1.1" 200 214974 "" "Prometheus/2.33.4" 2024-02-27T07:47:18.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:18 smithi149 ceph-mon[24796]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-02-27T07:47:18.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:18 smithi149 ceph-mon[29527]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-02-27T07:47:18.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:18 smithi195 ceph-mon[25709]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-02-27T07:47:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:20 smithi195 ceph-mon[25709]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:20.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:20 smithi149 ceph-mon[24796]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:20.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:20 smithi149 ceph-mon[29527]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:22 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:22 smithi195 ceph-mon[25709]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:22 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:22 smithi149 ceph-mon[29527]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:22 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:22 smithi149 ceph-mon[24796]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:23.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:23 smithi195 ceph-mon[25709]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:24.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:23 smithi149 ceph-mon[29527]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:24.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:23 smithi149 ceph-mon[24796]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:26.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:26 smithi149 ceph-mon[29527]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:26 smithi149 ceph-mon[24796]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:26.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:26 smithi195 ceph-mon[25709]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:26.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:27.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:27] "GET /metrics HTTP/1.1" 200 214931 "" "Prometheus/2.33.4" 2024-02-27T07:47:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:28 smithi195 ceph-mon[25709]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:28 smithi149 ceph-mon[29527]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:28 smithi149 ceph-mon[24796]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:30.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:30 smithi195 ceph-mon[25709]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:30.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:30 smithi149 ceph-mon[24796]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:30 smithi149 ceph-mon[29527]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:32.704 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:32 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:32.704 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:32 smithi149 ceph-mon[24796]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:32.704 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:32 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:32.704 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:32 smithi149 ceph-mon[29527]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:32 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:32 smithi195 ceph-mon[25709]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:34.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:33 smithi149 ceph-mon[24796]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:34.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:33 smithi149 ceph-mon[29527]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:34.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:33 smithi195 ceph-mon[25709]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:36.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:36 smithi195 ceph-mon[25709]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:36 smithi149 ceph-mon[24796]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:36.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:36 smithi149 ceph-mon[29527]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:37.928 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:37] "GET /metrics HTTP/1.1" 200 214931 "" "Prometheus/2.33.4" 2024-02-27T07:47:38.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:38 smithi195 ceph-mon[25709]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:38.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:38 smithi149 ceph-mon[29527]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:38.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:38 smithi149 ceph-mon[24796]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:39.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:39.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:47:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:40 smithi195 ceph-mon[25709]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:40 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:40.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:40 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:40.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[29527]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:40.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:40.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:40.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[24796]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:40.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:40.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:40 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:47:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:42 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:42 smithi195 ceph-mon[25709]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:42.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:42 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:42.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:42 smithi149 ceph-mon[24796]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:42.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:42 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:42.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:42 smithi149 ceph-mon[29527]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:44.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:43 smithi149 ceph-mon[24796]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:44.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:43 smithi149 ceph-mon[29527]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:44.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:43 smithi195 ceph-mon[25709]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:46.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:46 smithi195 ceph-mon[25709]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:46.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:46.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:46 smithi149 ceph-mon[24796]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:46.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:46 smithi149 ceph-mon[29527]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:47.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:47] "GET /metrics HTTP/1.1" 200 214886 "" "Prometheus/2.33.4" 2024-02-27T07:47:48.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:48 smithi195 ceph-mon[25709]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:48.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:48 smithi149 ceph-mon[24796]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:48.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:48 smithi149 ceph-mon[29527]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:50 smithi195 ceph-mon[25709]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:50.729 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:50 smithi149 ceph-mon[24796]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:50.729 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:50 smithi149 ceph-mon[29527]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:52 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:52.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:52 smithi195 ceph-mon[25709]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:52.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:52 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:52.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:52 smithi149 ceph-mon[29527]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:52 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:47:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:52 smithi149 ceph-mon[24796]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:54.076 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:53 smithi149 ceph-mon[24796]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:54.077 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:53 smithi195 ceph-mon[25709]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:54.077 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:53 smithi149 ceph-mon[29527]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:47:55.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:47:55.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:47:55.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:47:55.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:47:55.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:54 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:47:55.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:47:55.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:54 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:47:55.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:54 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:47:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:55 smithi149 ceph-mon[29527]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:56.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:55 smithi149 ceph-mon[24796]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:55 smithi195 ceph-mon[25709]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:47:56.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:47:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:47:57.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:47:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:47:57] "GET /metrics HTTP/1.1" 200 214885 "" "Prometheus/2.33.4" 2024-02-27T07:47:58.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:47:58 smithi195 ceph-mon[25709]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:47:58 smithi149 ceph-mon[29527]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:47:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:47:58 smithi149 ceph-mon[24796]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:00 smithi195 ceph-mon[25709]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:00.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:00 smithi149 ceph-mon[29527]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:00 smithi149 ceph-mon[24796]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:02 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:02 smithi195 ceph-mon[25709]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:02 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:02 smithi149 ceph-mon[29527]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:02 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:02 smithi149 ceph-mon[24796]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:03 smithi149 ceph-mon[29527]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:03 smithi149 ceph-mon[24796]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:03 smithi195 ceph-mon[25709]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:06.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:06 smithi195 ceph-mon[25709]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:06.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:06 smithi149 ceph-mon[29527]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:06.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:06 smithi149 ceph-mon[24796]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:06.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:07.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:07] "GET /metrics HTTP/1.1" 200 214885 "" "Prometheus/2.33.4" 2024-02-27T07:48:08.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:08 smithi195 ceph-mon[25709]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:08.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:08 smithi149 ceph-mon[29527]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:08.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:08 smithi149 ceph-mon[24796]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:10.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:10 smithi195 ceph-mon[25709]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:10.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:10 smithi149 ceph-mon[29527]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:10.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:10 smithi149 ceph-mon[24796]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:12.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:12 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:12 smithi195 ceph-mon[25709]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:12.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:12 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:12.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:12 smithi149 ceph-mon[29527]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:12 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:12 smithi149 ceph-mon[24796]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:14.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:13 smithi149 ceph-mon[29527]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:14.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:13 smithi149 ceph-mon[24796]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:13 smithi195 ceph-mon[25709]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:16.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:16 smithi195 ceph-mon[25709]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:16.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:16.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:16 smithi149 ceph-mon[29527]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:16.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:16 smithi149 ceph-mon[24796]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:17.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:17] "GET /metrics HTTP/1.1" 200 214875 "" "Prometheus/2.33.4" 2024-02-27T07:48:18.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:18 smithi195 ceph-mon[25709]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:18 smithi149 ceph-mon[24796]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:18.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:18 smithi149 ceph-mon[29527]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:20 smithi195 ceph-mon[25709]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:20.728 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:20 smithi149 ceph-mon[29527]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:20.728 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:20 smithi149 ceph-mon[24796]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:22 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:22 smithi195 ceph-mon[25709]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:22 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:22 smithi149 ceph-mon[29527]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:22 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:22 smithi149 ceph-mon[24796]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:24.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:23 smithi149 ceph-mon[29527]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:24.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:23 smithi149 ceph-mon[24796]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:23 smithi195 ceph-mon[25709]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:26.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:26 smithi195 ceph-mon[25709]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:26.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:26.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:26 smithi149 ceph-mon[29527]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:26.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:26 smithi149 ceph-mon[24796]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:27.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:27] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-02-27T07:48:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:28 smithi195 ceph-mon[25709]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:28 smithi149 ceph-mon[29527]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:28 smithi149 ceph-mon[24796]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:30.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:30 smithi195 ceph-mon[25709]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:30 smithi149 ceph-mon[29527]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:30.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:30 smithi149 ceph-mon[24796]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:32 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:32 smithi195 ceph-mon[25709]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:32.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:32 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:32.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:32 smithi149 ceph-mon[29527]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:32 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:32 smithi149 ceph-mon[24796]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:34.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:33 smithi149 ceph-mon[29527]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:34.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:33 smithi149 ceph-mon[24796]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:34.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:33 smithi195 ceph-mon[25709]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:36.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:36.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:36 smithi195 ceph-mon[25709]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:36 smithi149 ceph-mon[24796]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:36.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:36 smithi149 ceph-mon[29527]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:37.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:37] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-02-27T07:48:38.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:38 smithi195 ceph-mon[25709]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:38.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:38 smithi149 ceph-mon[29527]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:38.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:38 smithi149 ceph-mon[24796]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:48:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:48:40.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:40 smithi195 ceph-mon[25709]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:40.922 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:40 smithi149 ceph-mon[29527]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:40.922 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:40 smithi149 ceph-mon[24796]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:41.334 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-27T07:48:42.563 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:42 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:42.563 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:42 smithi149 ceph-mon[24796]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:42.563 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:42 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:42.563 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:42 smithi149 ceph-mon[29527]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:42 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:42.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:42 smithi195 ceph-mon[25709]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:43.819 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:43 smithi149 ceph-mon[29527]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:43.820 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:43 smithi149 ceph-mon[24796]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:43.844 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-27T07:48:44.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:43 smithi195 ceph-mon[25709]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:46.415 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-27T07:48:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:46 smithi195 ceph-mon[25709]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:46.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:46.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:46 smithi149 ceph-mon[29527]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:46.727 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:46 smithi149 ceph-mon[24796]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:47.771 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:47] "GET /metrics HTTP/1.1" 200 214898 "" "Prometheus/2.33.4" 2024-02-27T07:48:48.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:48 smithi195 ceph-mon[25709]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:48.738 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:48 smithi149 ceph-mon[24796]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:48.738 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:48 smithi149 ceph-mon[29527]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:48.762 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-27T07:48:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:50 smithi195 ceph-mon[25709]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:50.753 INFO:teuthology.orchestra.run.smithi149.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:50.754 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:50 smithi149 ceph-mon[24796]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:50.754 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:50 smithi149 ceph-mon[29527]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:51.294 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T07:48:51.305 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'lock': False, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'verbose': True, 'machine_type': 'smithi', 'os_type': None, 'owner': 'scheduled_adking@teuthology', 'interactive_on_error': False, 'suite_path': None, 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'os_version': None, 'block': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574539', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'mon.a': ['ceph config set mgr mgr/cephadm/use_repo_digest false --force']}}, {'cephadm.shell': {'mon.a': ['radosgw-admin realm create --rgw-realm=r --default', 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default', 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default', 'radosgw-admin period update --rgw-realm=r --commit', 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000', 'ceph orch apply rgw smpl', 'ceph osd pool create foo', 'rbd pool init foo', 'ceph orch apply iscsi foo u p', 'sleep 120', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'mon.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph versions', 'ceph orch upgrade status', 'ceph health detail', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1", "ceph orch ls | grep '^osd '"]}}, {'cephadm.shell': {'mon.a': ['ceph orch upgrade ls', 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0', 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']], [Remote(name='ubuntu@smithi195.front.sepia.ceph.com'), ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCskd1lovrDBBAA0xYDOJpP6SvAF/xPbbtVfw==\n', bootstrap_remote=Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '448a3a16-d543-11ee-95c3-87774f69a715', 'mon election default strategy': 1}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': 'true', 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 448a3a16-d543-11ee-95c3-87774f69a715\n[global]\n\tfsid = 448a3a16-d543-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]\n', first_mgr='y', first_mon='a', first_mon_role='mon.a', fsid='448a3a16-d543-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCrkd1ls4yDDxAAI5L1lY3CbYkl+aQJJduxzA==\n\tcaps mon = "allow *"\n', mons={'mon.a': '172.21.15.149', 'mon.c': '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'mon.b': '172.21.15.195'}, roleless=False, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'mon.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph versions', 'ceph orch upgrade status', 'ceph health detail', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1", "ceph orch ls | grep '^osd '"]}, 'cluster_name': ''} 2024-02-27T07:48:51.305 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T07:48:51.305 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-27T07:48:51.901 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:51.901 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:51.901 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:51.901 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:51.901 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[29527]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:48:51.902 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:51.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:51.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:51 smithi149 ceph-mon[24796]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:52.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:48:52.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:48:52.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:51 smithi195 ceph-mon[25709]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:53.461 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:48:54.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:53 smithi149 ceph-mon[29527]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:54.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:53 smithi149 ceph-mon[29527]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:54.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:53 smithi149 ceph-mon[24796]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:54.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:53 smithi149 ceph-mon[24796]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:54.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:53 smithi195 ceph-mon[25709]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:54.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:53 smithi195 ceph-mon[25709]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (2m) 2m ago 3m 14.6M - ba2b418f427c 78674f382d4a 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (2m) 2m ago 2m 47.8M - 8.3.5 dad864ee21e9 046d041033a6 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (2m) 2m ago 2m 70.9M - 3.5 e1d6a67b021e 05decc131ea6 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443 running (7m) 2m ago 7m 413M - 17.2.0 e1d6a67b021e 92be91019b9d 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 0d1c71eebda2 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (9m) 2m ago 9m 46.9M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (7m) 2m ago 7m 40.1M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:48:54.264 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (8m) 2m ago 8m 42.6M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (3m) 2m ago 3m 16.0M - 1dbe0e931976 fd34146eb0b6 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (3m) 2m ago 3m 16.2M - 1dbe0e931976 4082ac6575b8 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (7m) 2m ago 7m 48.8M 1198M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (6m) 2m ago 6m 49.6M 1198M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (6m) 2m ago 6m 45.2M 1198M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (6m) 2m ago 6m 48.0M 1198M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (5m) 2m ago 5m 48.0M 4272M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (5m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (5m) 2m ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (4m) 2m ago 4m 46.2M 4272M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (2m) 2m ago 3m 44.7M - 514e6a882f6e 0e171c4a47b3 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (2m) 2m ago 2m 68.3M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (2m) 2m ago 2m 68.2M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:48:54.265 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (2m) 2m ago 2m 69.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:48:54.747 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:48:54.748 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:48:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:54 smithi149 ceph-mon[24796]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:54 smithi149 ceph-mon[24796]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:54 smithi149 ceph-mon[29527]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:54 smithi149 ceph-mon[29527]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.180 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:48:55.180 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:48:55.180 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:48:55.181 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [], 2024-02-27T07:48:55.181 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "", 2024-02-27T07:48:55.181 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 image" 2024-02-27T07:48:55.181 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:48:55.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:54 smithi195 ceph-mon[25709]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:54 smithi195 ceph-mon[25709]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.656 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3926826314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[29527]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[29527]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4287256064' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3926826314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[24796]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[24796]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:55 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4287256064' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:48:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:55 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3926826314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:48:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:55 smithi195 ceph-mon[25709]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:48:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:55 smithi195 ceph-mon[25709]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:48:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:55 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4287256064' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:48:56.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:48:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:48:57.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:48:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:48:57] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-02-27T07:48:58.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:48:58 smithi195 ceph-mon[25709]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:58.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:48:58 smithi149 ceph-mon[29527]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:48:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:48:58 smithi149 ceph-mon[24796]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:00 smithi195 ceph-mon[25709]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:00.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:00 smithi149 ceph-mon[29527]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:00 smithi149 ceph-mon[24796]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:02 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:02.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:02 smithi195 ceph-mon[25709]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:02 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:02 smithi149 ceph-mon[29527]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:02 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:02 smithi149 ceph-mon[24796]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:03 smithi149 ceph-mon[29527]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:03 smithi149 ceph-mon[24796]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:03 smithi195 ceph-mon[25709]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:06.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:06 smithi195 ceph-mon[25709]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:06.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:49:06.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:06 smithi149 ceph-mon[29527]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:06.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:06 smithi149 ceph-mon[24796]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:07.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:07] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-02-27T07:49:08.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:08 smithi195 ceph-mon[25709]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:08.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:08 smithi149 ceph-mon[29527]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:08.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:08 smithi149 ceph-mon[24796]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:10 smithi195 ceph-mon[25709]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:10.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:10 smithi149 ceph-mon[24796]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:10.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:10 smithi149 ceph-mon[29527]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:12.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:12 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:12.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:12 smithi195 ceph-mon[25709]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:12 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:12.959 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:12 smithi149 ceph-mon[24796]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:12.959 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:12 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:12.959 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:12 smithi149 ceph-mon[29527]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:14.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:13 smithi149 ceph-mon[29527]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:14.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:13 smithi149 ceph-mon[24796]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:13 smithi195 ceph-mon[25709]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:16.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:16 smithi195 ceph-mon[25709]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:16.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:49:16.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:16 smithi149 ceph-mon[29527]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:16.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:16 smithi149 ceph-mon[24796]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:17.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:17] "GET /metrics HTTP/1.1" 200 214868 "" "Prometheus/2.33.4" 2024-02-27T07:49:18.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:18 smithi195 ceph-mon[25709]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:18.896 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:18 smithi149 ceph-mon[24796]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:18.896 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:18 smithi149 ceph-mon[29527]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[29527]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[24796]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:49:20.662 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:20 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:20 smithi195 ceph-mon[25709]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:20 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:20 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:49:20.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:20 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[29527]: Upgrade: Target is version 19.0.0-1736-g635ee685 (unknown) 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[29527]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, digests ['quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e'] 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[29527]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[29527]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 on smithi195 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[24796]: Upgrade: Target is version 19.0.0-1736-g635ee685 (unknown) 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[24796]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, digests ['quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e'] 2024-02-27T07:49:21.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[24796]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:49:21.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:21 smithi149 ceph-mon[24796]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 on smithi195 2024-02-27T07:49:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:21 smithi195 ceph-mon[25709]: Upgrade: Target is version 19.0.0-1736-g635ee685 (unknown) 2024-02-27T07:49:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:21 smithi195 ceph-mon[25709]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, digests ['quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e'] 2024-02-27T07:49:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:21 smithi195 ceph-mon[25709]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:49:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:21 smithi195 ceph-mon[25709]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 on smithi195 2024-02-27T07:49:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:22 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:22.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:22 smithi149 ceph-mon[29527]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:22 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:22.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:22 smithi149 ceph-mon[24796]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:22.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:22 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:22.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:22 smithi195 ceph-mon[25709]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:23 smithi149 ceph-mon[29527]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:23.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:23 smithi149 ceph-mon[24796]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:23.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:49:23.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:49:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:23 smithi195 ceph-mon[25709]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:25.981 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (3m) 2m ago 3m 14.6M - ba2b418f427c 78674f382d4a 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (3m) 2m ago 3m 47.8M - 8.3.5 dad864ee21e9 046d041033a6 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (2m) 2m ago 2m 70.9M - 3.5 e1d6a67b021e 05decc131ea6 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443 running (8m) 2m ago 8m 413M - 17.2.0 e1d6a67b021e 92be91019b9d 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 0d1c71eebda2 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (9m) 2m ago 9m 46.9M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (8m) 2m ago 8m 40.1M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (8m) 2m ago 8m 42.6M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (4m) 2m ago 4m 16.0M - 1dbe0e931976 fd34146eb0b6 2024-02-27T07:49:26.572 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (4m) 2m ago 4m 16.2M - 1dbe0e931976 4082ac6575b8 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (7m) 2m ago 7m 48.8M 1198M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (7m) 2m ago 7m 49.6M 1198M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (7m) 2m ago 7m 45.2M 1198M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (6m) 2m ago 6m 48.0M 1198M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (6m) 2m ago 6m 48.0M 4272M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (5m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (5m) 2m ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (5m) 2m ago 5m 46.2M 4272M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (3m) 2m ago 4m 44.7M - 514e6a882f6e 0e171c4a47b3 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (3m) 2m ago 3m 68.3M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (2m) 2m ago 2m 68.2M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:49:26.573 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (2m) 2m ago 2m 69.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:49:26.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:26 smithi195 ceph-mon[25709]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:26.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:49:26.919 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:49:26.920 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:26.921 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:49:26.921 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-02-27T07:49:26.921 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:49:26.921 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:49:26.921 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:26 smithi149 ceph-mon[29527]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:26.921 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:26 smithi149 ceph-mon[24796]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:27.216 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [], 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "0/23 daemons upgraded", 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 image on host smithi195" 2024-02-27T07:49:27.217 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:49:27.568 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:49:27.568 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[29527]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.568 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[29527]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.568 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/954815260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:27.568 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[24796]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.569 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[24796]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.569 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/954815260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:27.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:27] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-02-27T07:49:27.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:27 smithi195 ceph-mon[25709]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:27 smithi195 ceph-mon[25709]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:27.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:27 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/954815260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[24796]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[24796]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[24796]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1403414803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[29527]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[29527]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[29527]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:28 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1403414803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:49:28.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:28 smithi195 ceph-mon[25709]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:28 smithi195 ceph-mon[25709]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:28.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:28 smithi195 ceph-mon[25709]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:28.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:28 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1403414803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:49:30.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:30 smithi149 ceph-mon[24796]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:30 smithi149 ceph-mon[29527]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:30 smithi195 ceph-mon[25709]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:32 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:32.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:32 smithi149 ceph-mon[24796]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:32.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:32 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:32.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:32 smithi149 ceph-mon[29527]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:32.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:32 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:32.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:32 smithi195 ceph-mon[25709]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:33.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:33 smithi149 ceph-mon[24796]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:33.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:33 smithi149 ceph-mon[29527]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:33.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:49:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:49:33.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:49:33.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:49:34.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:33 smithi195 ceph-mon[25709]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:36 smithi195 ceph-mon[25709]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:36.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:49:36.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:36 smithi149 ceph-mon[29527]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:36 smithi149 ceph-mon[24796]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:37.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:37] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-02-27T07:49:38.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:38 smithi149 ceph-mon[29527]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:38.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:38 smithi149 ceph-mon[24796]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:38 smithi195 ceph-mon[25709]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:39.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:39 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:39.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:39 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:39.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:49:39.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:39 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:49:40.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:40 smithi149 ceph-mon[29527]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:40.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:40 smithi149 ceph-mon[24796]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:40 smithi195 ceph-mon[25709]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:42.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:42 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:42.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:42 smithi149 ceph-mon[29527]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:42.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:42 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:42.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:42 smithi149 ceph-mon[24796]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:42 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:42 smithi195 ceph-mon[25709]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:43.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:43 smithi149 ceph-mon[24796]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:43.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:43 smithi149 ceph-mon[29527]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:43.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:43 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:49:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:49:43.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:43 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:49:43.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:43 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:49:44.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:43 smithi195 ceph-mon[25709]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:46.670 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:46 smithi195 ceph-mon[25709]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:46.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:46 smithi149 ceph-mon[29527]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:46.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:46 smithi149 ceph-mon[24796]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:46.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[26895]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:49:47.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:47] "GET /metrics HTTP/1.1" 200 214841 "" "Prometheus/2.33.4" 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: Upgrade: Updating mgr.x 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: Deploying daemon mgr.x on smithi195 2024-02-27T07:49:48.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[29527]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: Upgrade: Updating mgr.x 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: Deploying daemon mgr.x on smithi195 2024-02-27T07:49:48.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:48 smithi149 ceph-mon[24796]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: Upgrade: Updating mgr.x 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:48.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: Deploying daemon mgr.x on smithi195 2024-02-27T07:49:48.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:48 smithi195 ceph-mon[25709]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:48.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:48 smithi195 systemd[1]: Stopping Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:49:48.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:48 smithi195 bash[46726]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr.x" found: no such container 2024-02-27T07:49:49.124 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:48 smithi195 podman[46736]: 2024-02-27 07:49:48.818362894 +0000 UTC m=+0.143454602 container died 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0) 2024-02-27T07:49:49.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 podman[46736]: 2024-02-27 07:49:49.222820009 +0000 UTC m=+0.547911720 container cleanup 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vendor=Red Hat, Inc., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-27T07:49:49.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 bash[46736]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x 2024-02-27T07:49:49.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 bash[46763]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr.x" found: no such container 2024-02-27T07:49:49.942 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 podman[46750]: 2024-02-27 07:49:49.532641648 +0000 UTC m=+0.706983191 container remove 92be91019b9d778fcb1991cf7d631cc03d5494883cb6ac685602262d4666cc03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, architecture=x86_64, GIT_BRANCH=HEAD, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc.) 2024-02-27T07:49:49.943 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-27T07:49:49.943 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-27T07:49:49.943 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 systemd[1]: Stopped Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:49:49.943 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Consumed 16.496s CPU time. 2024-02-27T07:49:49.943 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 systemd[1]: Starting Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:49:50.206 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 podman[46870]: 2024-02-27 07:49:49.941897068 +0000 UTC m=+0.112070789 container create 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:49:50.207 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:49 smithi195 podman[46870]: 2024-02-27 07:49:49.847318851 +0000 UTC m=+0.017492532 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:49:50.207 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 podman[46870]: 2024-02-27 07:49:50.178845439 +0000 UTC m=+0.349019120 container init 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-27T07:49:50.207 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 podman[46870]: 2024-02-27 07:49:50.181590414 +0000 UTC m=+0.351764100 container start 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True) 2024-02-27T07:49:50.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-mgr[46888]: -- 172.21.15.195:0/1958656509 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563da51fe5a0 con 0x563da51d3000 2024-02-27T07:49:50.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 bash[46870]: 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 2024-02-27T07:49:50.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 systemd[1]: Started Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:49:50.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[29527]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[24796]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:50 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:49:50.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-mon[25709]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:50.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:50.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:49:50.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:49:51.121 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:50.765+0000 7fd0b353d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:49:51.121 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:50.859+0000 7fd0b353d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:49:51.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:51.550+0000 7fd0b353d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:49:51.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:51.650+0000 7fd0b353d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:49:52.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:51.803+0000 7fd0b353d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:49:52.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:51.904+0000 7fd0b353d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:49:52.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:51.990+0000 7fd0b353d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:49:52.556 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:52.287+0000 7fd0b353d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:49:52.556 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:52.411+0000 7fd0b353d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:49:52.556 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:52.502+0000 7fd0b353d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:49:52.556 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:52.556 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-mon[25709]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:52.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:52 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:52.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:52 smithi149 ceph-mon[29527]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:52 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:49:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:52 smithi149 ceph-mon[24796]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:52.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:52.605+0000 7fd0b353d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:49:52.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:52 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:52.789+0000 7fd0b353d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:49:53.426 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:53.018Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T07:49:53.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-mon[29527]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:53.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-mon[24796]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:53.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:49:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T07:49:53.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T07:49:53.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:49:54.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:53 smithi195 ceph-mon[25709]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:54.070 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:53.808+0000 7fd0b353d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:49:54.070 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:53.942+0000 7fd0b353d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:49:54.388 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:54.291+0000 7fd0b353d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:49:54.388 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:54.387+0000 7fd0b353d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:49:54.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:54.599+0000 7fd0b353d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:49:55.065 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:54.796+0000 7fd0b353d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:49:55.383 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:55 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:55.100+0000 7fd0b353d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:49:55.634 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:55 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:55.634 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:55 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:55.634 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:55 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:55.382+0000 7fd0b353d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:49:55.634 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:55 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:55.624+0000 7fd0b353d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:49:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:55 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:55.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:55 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:55 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:55.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:55 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:49:56.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:56 smithi195 ceph-mon[25709]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:56.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:56.438+0000 7fd0b353d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:49:56.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:56 smithi149 ceph-mon[24796]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:56.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:56 smithi149 ceph-mon[29527]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:49:57.349 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:57.025+0000 7fd0b353d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:49:57.349 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:57.127+0000 7fd0b353d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:49:57.349 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:57.346+0000 7fd0b353d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: Standby manager daemon x restarted 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:49:57.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:49:57] ENGINE Bus STARTING 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: CherryPy Checker: 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: The Application mounted at '' has an empty config. 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:49:57] ENGINE Serving on http://:::9283 2024-02-27T07:49:57.688 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:49:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:49:57] ENGINE Bus STARTED 2024-02-27T07:49:57.886 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:49:57.492Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:49:57.886 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: Standby manager daemon x restarted 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:49:57] "GET /metrics HTTP/1.1" 200 214844 "" "Prometheus/2.33.4" 2024-02-27T07:49:57.887 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:49:57.887 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: Standby manager daemon x restarted 2024-02-27T07:49:57.888 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: Standby manager daemon x started 2024-02-27T07:49:57.888 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:49:57.888 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:49:57.888 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:49:57.888 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (3m) 3m ago 4m 14.6M - ba2b418f427c 78674f382d4a 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (3m) 4s ago 3m 55.1M - 8.3.5 dad864ee21e9 046d041033a6 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (3m) 3m ago 3m 70.9M - 3.5 e1d6a67b021e 05decc131ea6 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283 running (8s) 4s ago 8m 52.5M - 19.0.0-1736-g635ee685 c4eb0759a73a 94caded06864 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 0d1c71eebda2 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (10m) 3m ago 10m 46.9M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (8m) 4s ago 8m 37.8M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:49:58.469 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (9m) 3m ago 9m 42.6M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (4m) 3m ago 4m 16.0M - 1dbe0e931976 fd34146eb0b6 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (4m) 4s ago 4m 17.5M - 1dbe0e931976 4082ac6575b8 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (8m) 3m ago 8m 48.8M 1198M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (7m) 3m ago 7m 49.6M 1198M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (7m) 3m ago 7m 45.2M 1198M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (7m) 3m ago 7m 48.0M 1198M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (6m) 4s ago 6m 51.5M 4272M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (6m) 4s ago 6m 47.6M 4272M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (6m) 4s ago 6m 47.6M 4272M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (5m) 4s ago 5m 49.0M 4272M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (3m) 4s ago 5m 65.0M - 514e6a882f6e 0e171c4a47b3 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (3m) 3m ago 3m 68.3M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (3m) 4s ago 3m 69.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (3m) 3m ago 3m 70.2M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:49:58.470 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (3m) 4s ago 3m 70.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:49:58.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:58 smithi195 ceph-mon[25709]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:58.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:58 smithi195 ceph-mon[25709]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-27T07:49:58.816 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:49:58.817 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:49:58.818 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:49:58.818 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-27T07:49:58.818 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:49:58.818 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:49:58.818 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:49:58.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:58 smithi149 ceph-mon[24796]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:58.818 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:58 smithi149 ceph-mon[24796]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-27T07:49:58.818 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:58 smithi149 ceph-mon[29527]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:49:58.818 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:58 smithi149 ceph-mon[29527]: mgrmap e21: y(active, since 5m), standbys: x 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [], 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading mgr daemons" 2024-02-27T07:49:59.113 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:49:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:49:59.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:59 smithi195 ceph-mon[25709]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:59 smithi195 ceph-mon[25709]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:59 smithi195 ceph-mon[25709]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:59 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2814931523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:49:59 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2239339712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[29527]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[29527]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[29527]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2814931523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2239339712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:49:59.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[24796]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[24796]: from='client.25030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[24796]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:49:59.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2814931523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:49:59.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:49:59 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2239339712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: overall HEALTH_OK 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[24796]: osdmap e96: 8 total, 8 up, 8 in 2024-02-27T07:50:00.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: overall HEALTH_OK 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:00 smithi149 ceph-mon[29527]: osdmap e96: 8 total, 8 up, 8 in 2024-02-27T07:50:00.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:00.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:00.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: overall HEALTH_OK 2024-02-27T07:50:00.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' 2024-02-27T07:50:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='mgr.14433 172.21.15.149:0/1709073715' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-02-27T07:50:00.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:00 smithi195 ceph-mon[25709]: osdmap e96: 8 total, 8 up, 8 in 2024-02-27T07:50:01.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setuser ceph since I am not root 2024-02-27T07:50:01.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ignoring --setgroup ceph since I am not root 2024-02-27T07:50:01.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mgr[25033]: -- 172.21.15.149:0/1947365247 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a1bfc22340 con 0x55a1bfbfb000 2024-02-27T07:50:01.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE Bus STOPPING 2024-02-27T07:50:01.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T07:50:01.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE Bus STOPPED 2024-02-27T07:50:01.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE Bus STARTING 2024-02-27T07:50:01.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE Serving on http://:::9283 2024-02-27T07:50:01.465 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:50:01] ENGINE Bus STARTED 2024-02-27T07:50:01.902 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:01.614+0000 7f5ec36cc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:50:01.902 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:01.709+0000 7f5ec36cc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:50:01.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:50:01.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Failing over to other MGR 2024-02-27T07:50:01.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Standby manager daemon y started 2024-02-27T07:50:01.903 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: mgrmap e22: x(active, starting, since 0.999843s), standbys: y 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:50:01.904 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Manager daemon x is now available 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Queued rgw.foo for migration 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: Queued rgw.smpl for migration 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.905 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Failing over to other MGR 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Standby manager daemon y started 2024-02-27T07:50:01.906 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: mgrmap e22: x(active, starting, since 0.999843s), standbys: y 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Manager daemon x is now available 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Queued rgw.foo for migration 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: Queued rgw.smpl for migration 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:01.908 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.909 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.909 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:01.909 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:01 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Upgrade: Need to upgrade myself (mgr.y) 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Failing over to other MGR 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Standby manager daemon y started 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.14433 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: mgrmap e22: x(active, starting, since 0.999843s), standbys: y 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:50:01.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Manager daemon x is now available 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Queued rgw.foo for migration 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: Queued rgw.smpl for migration 2024-02-27T07:50:01.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:01.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:01 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-27T07:50:02.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.316+0000 7f5ec36cc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:50:02.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.406+0000 7f5ec36cc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:50:02.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.490+0000 7f5ec36cc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: Deploying cephadm binary to smithi195 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: Deploying cephadm binary to smithi149 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: [27/Feb/2024:07:50:02] ENGINE Bus STARTING 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: mgrmap e23: x(active, since 2s), standbys: y 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[24796]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.644 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.645 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: Deploying cephadm binary to smithi195 2024-02-27T07:50:02.645 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: Deploying cephadm binary to smithi149 2024-02-27T07:50:02.645 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: [27/Feb/2024:07:50:02] ENGINE Bus STARTING 2024-02-27T07:50:02.645 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: mgrmap e23: x(active, since 2s), standbys: y 2024-02-27T07:50:02.645 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-mon[29527]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:02.918 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.642+0000 7f5ec36cc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:50:02.918 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.723+0000 7f5ec36cc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:50:02.918 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.833+0000 7f5ec36cc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:50:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: Deploying cephadm binary to smithi195 2024-02-27T07:50:02.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: Deploying cephadm binary to smithi149 2024-02-27T07:50:02.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: [27/Feb/2024:07:50:02] ENGINE Bus STARTING 2024-02-27T07:50:02.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: mgrmap e23: x(active, since 2s), standbys: y 2024-02-27T07:50:02.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:02 smithi195 ceph-mon[25709]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:03.168 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:02.916+0000 7f5ec36cc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:50:03.168 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:03.081+0000 7f5ec36cc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:50:03.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:03.167+0000 7f5ec36cc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:50:03.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:50:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:50:03.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T07:50:03.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:03 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:03.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:50:04.358 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[29527]: [27/Feb/2024:07:50:02] ENGINE Serving on https://172.21.15.195:7150 2024-02-27T07:50:04.358 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[29527]: [27/Feb/2024:07:50:02] ENGINE Serving on http://172.21.15.195:8765 2024-02-27T07:50:04.358 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[29527]: [27/Feb/2024:07:50:02] ENGINE Bus STARTED 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[29527]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:04.225+0000 7f5ec36cc000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[24796]: [27/Feb/2024:07:50:02] ENGINE Serving on https://172.21.15.195:7150 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[24796]: [27/Feb/2024:07:50:02] ENGINE Serving on http://172.21.15.195:8765 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[24796]: [27/Feb/2024:07:50:02] ENGINE Bus STARTED 2024-02-27T07:50:04.359 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-mon[24796]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:04 smithi195 ceph-mon[25709]: [27/Feb/2024:07:50:02] ENGINE Serving on https://172.21.15.195:7150 2024-02-27T07:50:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:04 smithi195 ceph-mon[25709]: [27/Feb/2024:07:50:02] ENGINE Serving on http://172.21.15.195:8765 2024-02-27T07:50:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:04 smithi195 ceph-mon[25709]: [27/Feb/2024:07:50:02] ENGINE Bus STARTED 2024-02-27T07:50:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:04 smithi195 ceph-mon[25709]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:04.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:04.357+0000 7f5ec36cc000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:50:04.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:04.552+0000 7f5ec36cc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:50:04.644 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:04.642+0000 7f5ec36cc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:50:04.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:04.778+0000 7f5ec36cc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:50:05.422 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-mon[29527]: mgrmap e24: x(active, since 4s), standbys: y 2024-02-27T07:50:05.423 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-mon[24796]: mgrmap e24: x(active, since 4s), standbys: y 2024-02-27T07:50:05.423 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:05.107+0000 7f5ec36cc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:50:05.423 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:05.333+0000 7f5ec36cc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:50:05.423 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:05.421+0000 7f5ec36cc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:50:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:05 smithi195 ceph-mon[25709]: mgrmap e24: x(active, since 4s), standbys: y 2024-02-27T07:50:05.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:05.593+0000 7f5ec36cc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:50:06.365 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:06 smithi195 ceph-mon[25709]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:06 smithi149 ceph-mon[29527]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:06.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:06 smithi149 ceph-mon[24796]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:06.883 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:06 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:06.547+0000 7f5ec36cc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:50:06.883 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:06 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:06.881+0000 7f5ec36cc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:50:06.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:06] "GET /metrics HTTP/1.1" 200 35360 "" "Prometheus/2.33.4" 2024-02-27T07:50:07.140 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:06 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:06.980+0000 7f5ec36cc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:50:07.141 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:07.138+0000 7f5ec36cc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:50:07] ENGINE Bus STARTING 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: CherryPy Checker: 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: The Application mounted at '' has an empty config. 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:50:07] ENGINE Serving on http://:::9283 2024-02-27T07:50:07.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: [27/Feb/2024:07:50:07] ENGINE Bus STARTED 2024-02-27T07:50:07.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:50:08.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:08.029Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: Standby manager daemon y restarted 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: Standby manager daemon y started 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:50:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: Standby manager daemon y restarted 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: Standby manager daemon y started 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:50:08.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:08 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: Standby manager daemon y restarted 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: Standby manager daemon y started 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:50:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:50:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:08 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2818295173' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:50:09.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:09 smithi149 ceph-mon[29527]: mgrmap e25: x(active, since 8s), standbys: y 2024-02-27T07:50:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:09 smithi149 ceph-mon[24796]: mgrmap e25: x(active, since 8s), standbys: y 2024-02-27T07:50:09.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:09 smithi195 ceph-mon[25709]: mgrmap e25: x(active, since 8s), standbys: y 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[24796]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[29527]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:10 smithi195 ceph-mon[25709]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:50:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]': finished 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[24796]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:50:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]': finished 2024-02-27T07:50:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:12 smithi149 ceph-mon[29527]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]': finished 2024-02-27T07:50:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:12 smithi195 ceph-mon[25709]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:50:13.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:13 smithi149 ceph-mon[24796]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:50:13.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:13 smithi149 ceph-mon[29527]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:50:13.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:50:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:50:13.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:50:13.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:50:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:13 smithi195 ceph-mon[25709]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:50:16.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:16 smithi149 ceph-mon[24796]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:16 smithi149 ceph-mon[29527]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:16 smithi195 ceph-mon[25709]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:16.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:16] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.33.4" 2024-02-27T07:50:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:17 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:17 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:17 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:17.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[29527]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[24796]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:18 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:18.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:18 smithi195 ceph-mon[25709]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:50:18.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:18 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:18.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:18 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[29527]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:19 smithi149 ceph-mon[24796]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:19.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:19 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:19 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:19.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:19 smithi195 ceph-mon[25709]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:22 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:22 smithi195 ceph-mon[25709]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:22 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:22 smithi149 ceph-mon[24796]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:22.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:22 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:22.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:22 smithi149 ceph-mon[29527]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:22.595 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.195:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:50:23.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=error ts=2024-02-27T07:50:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:50:23.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.195:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.195 because it doesn't contain any IP SANs" 2024-02-27T07:50:23.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=warn ts=2024-02-27T07:50:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-02-27T07:50:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Adjusting osd_memory_target on smithi195 to 2992M 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:50:23.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Adjusting osd_memory_target on smithi195 to 2992M 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:23.930 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:23 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:23.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Adjusting osd_memory_target on smithi195 to 2992M 2024-02-27T07:50:23.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:23.966 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:23 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:50:24.677 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 systemd[1]: Stopping Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:24.677 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 bash[58384]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.a" found: no such container 2024-02-27T07:50:24.677 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 podman[58396]: 2024-02-27 07:50:24.458543568 +0000 UTC m=+0.130399821 container died fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:24.972 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 podman[58396]: 2024-02-27 07:50:24.931554554 +0000 UTC m=+0.603410813 container cleanup fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:24.972 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 bash[58396]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a 2024-02-27T07:50:24.972 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:24 smithi149 bash[58422]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.a" found: no such container 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.973 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[24796]: Reconfiguring daemon node-exporter.a on smithi149 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.974 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:24.975 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-27T07:50:24.975 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:24 smithi149 ceph-mon[29527]: Reconfiguring daemon node-exporter.a on smithi149 2024-02-27T07:50:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-27T07:50:25.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:24 smithi195 ceph-mon[25709]: Reconfiguring daemon node-exporter.a on smithi149 2024-02-27T07:50:25.615 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 podman[58411]: 2024-02-27 07:50:25.258324003 +0000 UTC m=+0.803655970 container remove fd34146eb0b6e223221a38aa5c06fe86f5fde490cf962b4c2cd26af33b0a04a4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: Stopped Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Consumed 2.682s CPU time. 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: Starting Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 podman[58493]: 2024-02-27 07:50:25.393134757 +0000 UTC m=+0.018947414 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-27T07:50:25.616 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 podman[58493]: 2024-02-27 07:50:25.492399644 +0000 UTC m=+0.118212288 container create 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-mon[29527]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-mon[24796]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 podman[58493]: 2024-02-27 07:50:25.728568458 +0000 UTC m=+0.354381104 container init 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 podman[58493]: 2024-02-27 07:50:25.731227964 +0000 UTC m=+0.357040610 container start 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.734Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.734Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.734Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.734Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=arp 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-27T07:50:26.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=edac 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-27T07:50:26.179 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=os 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-27T07:50:26.180 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=stat 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=time 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=uname 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-27T07:50:26.181 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-27T07:50:26.182 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-27T07:50:26.182 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a[58508]: ts=2024-02-27T07:50:25.735Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-27T07:50:26.182 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 bash[58493]: 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a 2024-02-27T07:50:26.182 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 07:50:25 smithi149 systemd[1]: Started Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:25 smithi195 ceph-mon[25709]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:26.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:26 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:26.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:26 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:26.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:26 smithi195 ceph-mon[25709]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:50:26.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:26 smithi195 ceph-mon[25709]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:50:26.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:26] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.33.4" 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[29527]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[29527]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[24796]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-27T07:50:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:26 smithi149 ceph-mon[24796]: Reconfiguring daemon alertmanager.a on smithi149 2024-02-27T07:50:27.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:27 smithi149 ceph-mon[29527]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:27.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:50:27.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:27 smithi149 ceph-mon[24796]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:28.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:27 smithi195 ceph-mon[25709]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:29.816 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:50:30.330 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:30 smithi149 ceph-mon[24796]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:30.330 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:30 smithi149 ceph-mon[29527]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:30 smithi195 ceph-mon[25709]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:30.480 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:50:30.480 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (4m) 20s ago 4m 26.6M - 0.23.0 ba2b418f427c 78674f382d4a 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (4m) 13s ago 4m 55.3M - 8.3.5 dad864ee21e9 046d041033a6 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (3m) 20s ago 3m 54.0M - 3.5 e1d6a67b021e 05decc131ea6 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283 running (40s) 13s ago 9m 494M - 19.0.0-1736-g635ee685 c4eb0759a73a 94caded06864 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:9283 running (10m) 20s ago 10m 381M - 17.2.0 e1d6a67b021e 0d1c71eebda2 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (10m) 20s ago 11m 52.9M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (9m) 13s ago 9m 43.1M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (9m) 20s ago 9m 47.1M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 starting - - - - 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (5m) 13s ago 5m 17.0M - 1.3.1 1dbe0e931976 4082ac6575b8 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (8m) 20s ago 8m 51.0M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (8m) 20s ago 8m 51.9M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:50:30.481 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (8m) 20s ago 8m 46.6M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (7m) 20s ago 7m 50.7M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (7m) 13s ago 7m 51.9M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (7m) 13s ago 6m 48.2M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (6m) 13s ago 6m 47.9M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (6m) 13s ago 6m 49.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (4m) 13s ago 5m 65.0M - 2.33.4 514e6a882f6e 0e171c4a47b3 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (4m) 20s ago 4m 69.8M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (3m) 13s ago 3m 69.9M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (3m) 20s ago 3m 71.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:50:30.482 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (3m) 13s ago 3m 71.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T07:50:30.875 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:50:30.876 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [], 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "", 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:50:31.214 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:50:31.419 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[24796]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.419 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[24796]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.419 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1904269004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:50:31.419 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[29527]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.419 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[29527]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.420 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:31 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1904269004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:50:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:31 smithi195 ceph-mon[25709]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:31 smithi195 ceph-mon[25709]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:31 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1904269004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:50:31.607 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2696469554' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:32.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:32.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:32.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2696469554' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:50:32.464 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.195:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:50:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: from='client.14892 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:50:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:32.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:50:32.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:32 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2696469554' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:50:32.948 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:32 smithi149 systemd[1]: Stopping Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:32.948 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:32 smithi149 bash[58890]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager.a" found: no such container 2024-02-27T07:50:33.233 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:32 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[49459]: level=info ts=2024-02-27T07:50:32.947Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-27T07:50:33.233 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:32 smithi149 podman[58901]: 2024-02-27 07:50:32.978115959 +0000 UTC m=+0.140367828 container died 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:33.673 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 podman[58901]: 2024-02-27 07:50:33.414587586 +0000 UTC m=+0.576839461 container cleanup 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:33.673 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 bash[58901]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a 2024-02-27T07:50:33.714 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:33.506Z caller=notifier.go:526 level=error component=notifier alertmanager=http://172.21.15.149:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"http://172.21.15.149:9093/api/v2/alerts\": dial tcp 172.21.15.149:9093: connect: connection refused" 2024-02-27T07:50:33.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:33 smithi149 ceph-mon[24796]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:33.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:33 smithi149 ceph-mon[29527]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:33.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 podman[58915]: 2024-02-27 07:50:33.673111733 +0000 UTC m=+0.693180984 container remove 78674f382d4a0943288c7447a1207cd485a06afa7afebc2d44cceaa4a2050676 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:33.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 podman[58915]: 2024-02-27 07:50:33.77543769 +0000 UTC m=+0.795506952 volume remove 6ba6d3e1a02a54d8fe6c69fe3a9ebf44b0d3649aada7b0d73e5f819098da27da 2024-02-27T07:50:33.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 bash[58927]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager.a" found: no such container 2024-02-27T07:50:33.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-27T07:50:33.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 systemd[1]: Stopped Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:33.928 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:33 smithi149 systemd[1]: Starting Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:34.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:33 smithi195 ceph-mon[25709]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:34.425 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 podman[59006]: 2024-02-27 07:50:34.007444272 +0000 UTC m=+0.019439036 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-27T07:50:34.425 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 podman[59006]: 2024-02-27 07:50:34.126358251 +0000 UTC m=+0.138353001 volume create 300b194248c0b6c4933a95a8f062a503b8c06cf89b8a3b3797564d2cdc70ec2d 2024-02-27T07:50:34.425 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 podman[59006]: 2024-02-27 07:50:34.218531765 +0000 UTC m=+0.230526515 container create 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 podman[59006]: 2024-02-27 07:50:34.521092001 +0000 UTC m=+0.533086750 container init 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 podman[59006]: 2024-02-27 07:50:34.523738919 +0000 UTC m=+0.535733675 container start 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.534Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.534Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.535Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.149 port=9094 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.536Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-27T07:50:34.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.562Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:50:34.678 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.562Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-27T07:50:34.678 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.564Z caller=main.go:518 msg=Listening address=:9093 2024-02-27T07:50:34.678 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:34.564Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-27T07:50:34.678 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 bash[59006]: 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd 2024-02-27T07:50:34.678 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:34 smithi149 systemd[1]: Started Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:50:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[29527]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:50:35.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:35 smithi149 ceph-mon[24796]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:50:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:50:35.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:50:35.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:35 smithi195 ceph-mon[25709]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:36.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:36.537Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000961232s 2024-02-27T07:50:36.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:36] "GET /metrics HTTP/1.1" 200 38159 "" "Prometheus/2.33.4" 2024-02-27T07:50:37.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:37 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:50:38.199 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:38 smithi149 ceph-mon[29527]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:38.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:38 smithi149 ceph-mon[24796]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:38.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-mon[25709]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:38.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:50:38.309+0000 7fd0391ed700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Creating ceph-iscsi config... 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return _run_code(code, main_globals, None, 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: exec(code, run_globals) 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:38.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:38.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return self.event_loop.get_result(coro, timeout) 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return future.result(timeout) 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return self.__get_result() 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: raise self._exception 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-27T07:50:38.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: config_blobs=daemon_spec.final_config, 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Creating ceph-iscsi config... 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return _run_code(code, main_globals, None, 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:38.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: exec(code, run_globals) 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:38.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[24796]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: exec(code, run_globals) 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.428 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: return future.result(timeout) 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: return self.__get_result() 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: raise self._exception 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: config_blobs=daemon_spec.final_config, 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-27T07:50:40.429 INFO:journalctl@ceph.mon.a.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: exec(code, run_globals) 2024-02-27T07:50:40.430 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[29527]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.431 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: exec(code, run_globals) 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.432 INFO:journalctl@ceph.mon.c.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: return future.result(timeout) 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: return self.__get_result() 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: raise self._exception 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: config_blobs=daemon_spec.final_config, 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-27T07:50:40.433 INFO:journalctl@ceph.mon.c.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: exec(code, run_globals) 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.434 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[24796]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[24796]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[24796]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:40.435 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[29527]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[29527]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:50:40.436 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:40 smithi149 ceph-mon[29527]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:40 smithi195 ceph-mon[25709]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: exec(code, run_globals) 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.465 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: return future.result(timeout) 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.__get_result() 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-27T07:50:40.466 INFO:journalctl@ceph.mon.b.smithi195.stdout: raise self._exception 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: config_blobs=daemon_spec.final_config, 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: Creating ceph-iscsi config... 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: return _run_code(code, main_globals, None, 2024-02-27T07:50:40.467 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: exec(code, run_globals) 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:40 smithi195 ceph-mon[25709]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:40 smithi195 ceph-mon[25709]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:50:40.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:40 smithi195 ceph-mon[25709]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:50:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:42 smithi149 ceph-mon[29527]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:42 smithi149 ceph-mon[24796]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:42 smithi195 ceph-mon[25709]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:42.464 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:42.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.195:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:50:44.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:43 smithi149 ceph-mon[29527]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:44.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:43 smithi149 ceph-mon[24796]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:44.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:43 smithi195 ceph-mon[25709]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:50:44.926 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 07:50:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T07:50:44.539Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003067713s 2024-02-27T07:50:46.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:46 smithi149 ceph-mon[29527]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:46 smithi149 ceph-mon[24796]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:46 smithi195 ceph-mon[25709]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:46.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:46 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:46] "GET /metrics HTTP/1.1" 200 38159 "" "Prometheus/2.33.4" 2024-02-27T07:50:47.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:47 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:47 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:47 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:50:47.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:50:47 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[25029]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:50:48.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:48 smithi149 ceph-mon[29527]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:48.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:48 smithi149 ceph-mon[24796]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:48.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:48 smithi195 ceph-mon[25709]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:50.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:50 smithi149 ceph-mon[29527]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:50.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:50 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/248189745' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:50:50.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:50 smithi149 ceph-mon[24796]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:50.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:50 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/248189745' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:50:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-mon[25709]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-27T07:50:50.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/248189745' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 systemd[1]: Stopping Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 bash[50392]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.277Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-27T07:50:50.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-27T07:50:50.466 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-27T07:50:50.466 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-27T07:50:50.466 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[42611]: ts=2024-02-27T07:50:50.278Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-27T07:50:50.466 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 podman[50405]: 2024-02-27 07:50:50.287945089 +0000 UTC m=+0.139426439 container died 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:50.887 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 podman[50405]: 2024-02-27 07:50:50.643466915 +0000 UTC m=+0.494948303 container cleanup 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:50.887 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 bash[50405]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a 2024-02-27T07:50:50.887 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 bash[50431]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:50:51.154 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 podman[50419]: 2024-02-27 07:50:50.886462359 +0000 UTC m=+0.600454197 container remove 0e171c4a47b390c8b73a8e23e9daf69066a6da13e4949677e860bb6a63fdcf27 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:51.154 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-27T07:50:51.154 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 systemd[1]: Stopped Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:51.154 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service: Consumed 1.511s CPU time. 2024-02-27T07:50:51.155 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:50 smithi195 systemd[1]: Starting Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:51.155 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 podman[50498]: 2024-02-27 07:50:51.030960804 +0000 UTC m=+0.018639758 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-27T07:50:51.155 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 podman[50498]: 2024-02-27 07:50:51.154135347 +0000 UTC m=+0.141814300 container create 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:51.424 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]: dispatch 2024-02-27T07:50:51.425 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 podman[50498]: 2024-02-27 07:50:51.398295275 +0000 UTC m=+0.385974228 container init 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:51.425 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 podman[50498]: 2024-02-27 07:50:51.400793948 +0000 UTC m=+0.388472902 container start 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:50:51.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:51 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]: dispatch 2024-02-27T07:50:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:51 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]: dispatch 2024-02-27T07:50:51.714 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi195 (none))" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.424Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.426Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.426Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.427Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.429Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.429Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.861µs 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.429Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 bash[50498]: 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 2024-02-27T07:50:51.715 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 systemd[1]: Started Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:51.716 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.605Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.825Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.825Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.825Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.581µs wal_replay_duration=396.09923ms total_replay_duration=396.139758ms 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.830Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.830Z caller=main.go:947 level=info msg="TSDB started" 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.830Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.857Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.401723ms db_storage=853ns remote_storage=1.752µs web_handler=414ns query_engine=1.079µs scrape=571.748µs scrape_sd=158.171µs notify=18.323µs notify_sd=16.72µs rules=25.257949ms 2024-02-27T07:50:52.082 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:51.857Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]': finished 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: osdmap e97: 8 total, 8 up, 8 in 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]: dispatch 2024-02-27T07:50:52.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:52 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.433 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: Stopping Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:52.433 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 bash[50623]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.b" found: no such container 2024-02-27T07:50:52.434 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 podman[50636]: 2024-02-27 07:50:52.22890753 +0000 UTC m=+0.115881653 container died 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:52.544 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]': finished 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: osdmap e97: 8 total, 8 up, 8 in 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]: dispatch 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/3614944193' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2305001970"}]': finished 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: osdmap e97: 8 total, 8 up, 8 in 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]: dispatch 2024-02-27T07:50:52.545 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:52 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:52.714 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 podman[50636]: 2024-02-27 07:50:52.610097264 +0000 UTC m=+0.497071393 container cleanup 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:52.714 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 bash[50636]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b 2024-02-27T07:50:52.714 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 bash[50663]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.b" found: no such container 2024-02-27T07:50:53.095 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 podman[50650]: 2024-02-27 07:50:52.861446372 +0000 UTC m=+0.633456119 container remove 4082ac6575b86987ba3cb35766daa20ceb600bb5708ef0ce21e3bc2219512d27 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:53.096 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-27T07:50:53.096 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-27T07:50:53.096 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: Stopped Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:53.096 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Consumed 2.748s CPU time. 2024-02-27T07:50:53.096 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:52 smithi195 systemd[1]: Starting Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:50:53.371 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:53 smithi195 podman[50741]: 2024-02-27 07:50:52.99769042 +0000 UTC m=+0.017928907 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-27T07:50:53.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-27T07:50:53.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: Reconfiguring daemon node-exporter.b on smithi195 2024-02-27T07:50:53.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]': finished 2024-02-27T07:50:53.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: osdmap e98: 8 total, 8 up, 8 in 2024-02-27T07:50:53.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4197524021' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:53 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.506 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-27T07:50:53.506 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: Reconfiguring daemon node-exporter.b on smithi195 2024-02-27T07:50:53.506 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]': finished 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: osdmap e98: 8 total, 8 up, 8 in 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4197524021' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: Reconfiguring daemon node-exporter.b on smithi195 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/293924246' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/784752624"}]': finished 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: osdmap e98: 8 total, 8 up, 8 in 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4197524021' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:53 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]: dispatch 2024-02-27T07:50:53.714 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:53 smithi195 podman[50741]: 2024-02-27 07:50:53.371173883 +0000 UTC m=+0.391412358 container create 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-mon[25709]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-27T07:50:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]': finished 2024-02-27T07:50:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-mon[25709]: osdmap e99: 8 total, 8 up, 8 in 2024-02-27T07:50:54.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1927196826' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:54.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[29527]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-27T07:50:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]': finished 2024-02-27T07:50:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[29527]: osdmap e99: 8 total, 8 up, 8 in 2024-02-27T07:50:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1927196826' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:54.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[24796]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-27T07:50:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/784752624"}]': finished 2024-02-27T07:50:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[24796]: osdmap e99: 8 total, 8 up, 8 in 2024-02-27T07:50:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1927196826' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:54.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:54 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]: dispatch 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 podman[50741]: 2024-02-27 07:50:54.758592686 +0000 UTC m=+1.778831164 container init 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 podman[50741]: 2024-02-27 07:50:54.76188129 +0000 UTC m=+1.782119769 container start 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.767Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.767Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=arp 2024-02-27T07:50:55.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.768Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-27T07:50:55.216 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=edac 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-27T07:50:55.217 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=os 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=stat 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-27T07:50:55.218 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=time 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=uname 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b[50759]: ts=2024-02-27T07:50:54.769Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 bash[50741]: 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d 2024-02-27T07:50:55.219 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 07:50:54 smithi195 systemd[1]: Started Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]': finished 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: osdmap e100: 8 total, 8 up, 8 in 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: Reconfiguring grafana.a (dependencies changed)... 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: Reconfiguring daemon grafana.a on smithi195 2024-02-27T07:50:56.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1911195397' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[29527]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]': finished 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: osdmap e100: 8 total, 8 up, 8 in 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: Reconfiguring grafana.a (dependencies changed)... 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: Reconfiguring daemon grafana.a on smithi195 2024-02-27T07:50:56.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1911195397' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:55 smithi149 ceph-mon[24796]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/4098348216"}]': finished 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: osdmap e100: 8 total, 8 up, 8 in 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: Reconfiguring grafana.a (dependencies changed)... 2024-02-27T07:50:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: Reconfiguring daemon grafana.a on smithi195 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1911195397' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]: dispatch 2024-02-27T07:50:56.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:55 smithi195 ceph-mon[25709]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:50:57.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[24796]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]': finished 2024-02-27T07:50:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[24796]: osdmap e101: 8 total, 8 up, 8 in 2024-02-27T07:50:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]: dispatch 2024-02-27T07:50:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[29527]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]': finished 2024-02-27T07:50:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[29527]: osdmap e101: 8 total, 8 up, 8 in 2024-02-27T07:50:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:56 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]: dispatch 2024-02-27T07:50:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:56 smithi195 ceph-mon[25709]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2945015925"}]': finished 2024-02-27T07:50:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:56 smithi195 ceph-mon[25709]: osdmap e101: 8 total, 8 up, 8 in 2024-02-27T07:50:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:56 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]: dispatch 2024-02-27T07:50:58.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:50:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:50:57] "GET /metrics HTTP/1.1" 200 38159 "" "Prometheus/2.33.4" 2024-02-27T07:50:58.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:58 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]': finished 2024-02-27T07:50:58.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:58 smithi195 ceph-mon[25709]: osdmap e102: 8 total, 8 up, 8 in 2024-02-27T07:50:58.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:58 smithi195 ceph-mon[25709]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:58.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:58 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]: dispatch 2024-02-27T07:50:58.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]': finished 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[24796]: osdmap e102: 8 total, 8 up, 8 in 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[24796]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]: dispatch 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2649726811' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/936127808"}]': finished 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[29527]: osdmap e102: 8 total, 8 up, 8 in 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[29527]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:50:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:58 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]: dispatch 2024-02-27T07:50:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:50:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:50:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:50:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:59 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]': finished 2024-02-27T07:50:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:50:59 smithi149 ceph-mon[29527]: osdmap e103: 8 total, 8 up, 8 in 2024-02-27T07:50:59.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:59 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]': finished 2024-02-27T07:50:59.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:50:59 smithi149 ceph-mon[24796]: osdmap e103: 8 total, 8 up, 8 in 2024-02-27T07:50:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:59 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/4194149098' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1184246026"}]': finished 2024-02-27T07:50:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:50:59 smithi195 ceph-mon[25709]: osdmap e103: 8 total, 8 up, 8 in 2024-02-27T07:51:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:00 smithi149 ceph-mon[29527]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:00 smithi149 ceph-mon[24796]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:00 smithi195 ceph-mon[25709]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:01 smithi149 ceph-mon[29527]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:01.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:01 smithi149 ceph-mon[24796]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:01 smithi195 ceph-mon[25709]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:01.956 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:51:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:02 smithi149 ceph-mon[29527]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:02 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:02.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:02 smithi149 ceph-mon[24796]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:02.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:02 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:02 smithi195 ceph-mon[25709]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:02.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:02 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:02.596 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 starting - - - - 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (5m) 45s ago 5m 55.3M - 8.3.5 dad864ee21e9 046d041033a6 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (4m) 52s ago 4m 54.0M - 3.5 e1d6a67b021e 05decc131ea6 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283 running (72s) 45s ago 9m 494M - 19.0.0-1736-g635ee685 c4eb0759a73a 94caded06864 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:9283 running (11m) 52s ago 11m 381M - 17.2.0 e1d6a67b021e 0d1c71eebda2 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (11m) 52s ago 11m 52.9M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (9m) 45s ago 9m 43.1M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (10m) 52s ago 10m 47.1M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 starting - - - - 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 starting - - - - 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (9m) 52s ago 9m 51.0M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (9m) 52s ago 9m 51.9M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (8m) 52s ago 8m 46.6M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (8m) 52s ago 8m 50.7M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:51:02.597 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (7m) 45s ago 7m 51.9M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (7m) 45s ago 7m 48.2M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (7m) 45s ago 7m 47.9M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (6m) 45s ago 6m 49.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 starting - - - - 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (4m) 52s ago 4m 69.8M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (4m) 45s ago 4m 69.9M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (4m) 52s ago 4m 71.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:51:02.598 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (4m) 45s ago 4m 71.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:51:02.984 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:51:02.984 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:02.985 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:51:02.986 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-02-27T07:51:02.986 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:51:02.986 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:51:02.986 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:51:03.314 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [], 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "", 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:51:03.315 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:51:03.315 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:03 smithi149 ceph-mon[24796]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:03.315 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:03 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1517745890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:03.316 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:03 smithi149 ceph-mon[29527]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:03.316 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:03 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1517745890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:03 smithi195 ceph-mon[25709]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:03 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1517745890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:03.707 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[24796]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[24796]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[24796]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1261007779' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[29527]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[29527]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[29527]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-27T07:51:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:04 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1261007779' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:04 smithi195 ceph-mon[25709]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:04 smithi195 ceph-mon[25709]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:04 smithi195 ceph-mon[25709]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-27T07:51:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:04 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1261007779' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:05.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:05 smithi149 ceph-mon[29527]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:05.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:05 smithi149 ceph-mon[24796]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:05 smithi195 ceph-mon[25709]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:06.155 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:06 smithi195 ceph-mon[25709]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:06.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:06 smithi149 ceph-mon[29527]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:06.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:06 smithi149 ceph-mon[24796]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:08.055 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:51:07] "GET /metrics HTTP/1.1" 200 38156 "" "Prometheus/2.33.4" 2024-02-27T07:51:08.305 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 systemd[1]: Stopping Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:08.305 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 bash[50954]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana.a" found: no such container 2024-02-27T07:51:08.305 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:51:08+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-27T07:51:08.305 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[41808]: t=2024-02-27T07:51:08+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-27T07:51:08.305 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 podman[50967]: 2024-02-27 07:51:08.223310212 +0000 UTC m=+0.129117982 container died 046d041033a64fe3156e433b12317d86e7f08a1fa6d2f8a5cc1e550a211b4c2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, description=Ceph Grafana Container, version=8.5, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-27T07:51:08.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:08 smithi195 ceph-mon[25709]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:51:08.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:08 smithi149 ceph-mon[29527]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:51:08.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:08 smithi149 ceph-mon[24796]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:51:08.862 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:51:08.863 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 podman[50967]: 2024-02-27 07:51:08.602220786 +0000 UTC m=+0.508028552 container cleanup 046d041033a64fe3156e433b12317d86e7f08a1fa6d2f8a5cc1e550a211b4c2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.expose-services=, version=8.5, architecture=x86_64) 2024-02-27T07:51:08.863 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 bash[50967]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a 2024-02-27T07:51:08.863 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 bash[50996]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana.a" found: no such container 2024-02-27T07:51:09.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 podman[50981]: 2024-02-27 07:51:08.861890774 +0000 UTC m=+0.636797963 container remove 046d041033a64fe3156e433b12317d86e7f08a1fa6d2f8a5cc1e550a211b4c2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.openshift.expose-services=, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, architecture=x86_64, io.buildah.version=1.24.2, release=236.1648460182) 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 systemd[1]: Stopped Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@grafana.a.service: Consumed 2.563s CPU time. 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:08 smithi195 systemd[1]: Starting Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 podman[51071]: 2024-02-27 07:51:08.99030903 +0000 UTC m=+0.017328165 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-27T07:51:09.215 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 podman[51071]: 2024-02-27 07:51:09.096138645 +0000 UTC m=+0.123157760 container create 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.buildah.version=1.24.2, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, architecture=x86_64, io.openshift.expose-services=, name=ubi8, release=236.1648460182) 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 podman[51071]: 2024-02-27 07:51:09.390597076 +0000 UTC m=+0.417616204 container init 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, name=ubi8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, version=8.5, architecture=x86_64) 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 podman[51071]: 2024-02-27 07:51:09.39330489 +0000 UTC m=+0.420324021 container start 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , distribution-scope=public, vcs-type=git, version=8.5, io.buildah.version=1.24.2, architecture=x86_64, name=ubi8, description=Ceph Grafana Container, io.openshift.expose-services=, release=236.1648460182, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762) 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-27T07:51:09.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="App mode production" logger=settings 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-27T07:51:09.478 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=613.405µs 2024-02-27T07:51:09.479 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-27T07:51:09.479 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-27T07:51:09.479 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-27T07:51:09.479 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 bash[51071]: 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 2024-02-27T07:51:09.479 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 systemd[1]: Started Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-27T07:51:09.767 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T07:51:09+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPING 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPED 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTING 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Serving on http://:::9283 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTED 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPING 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPED 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTING 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Serving on http://:::9283 2024-02-27T07:51:09.768 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTED 2024-02-27T07:51:10.167 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPING 2024-02-27T07:51:10.167 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T07:51:10.167 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STOPPED 2024-02-27T07:51:10.168 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTING 2024-02-27T07:51:10.168 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Serving on http://:::9283 2024-02-27T07:51:10.168 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:09] ENGINE Bus STARTED 2024-02-27T07:51:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-27T07:51:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-27T07:51:10.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:10 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:10.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:10 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[24796]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:11 smithi149 ceph-mon[29527]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T07:51:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi149.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T07:51:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T07:51:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T07:51:11.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T07:51:11.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-27T07:51:11.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:11 smithi195 ceph-mon[25709]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:12.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:12.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: Upgrade: Updating mgr.y 2024-02-27T07:51:12.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[24796]: Deploying daemon mgr.y on smithi149 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: Upgrade: Updating mgr.y 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:12.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:12 smithi149 ceph-mon[29527]: Deploying daemon mgr.y on smithi149 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: Upgrade: Updating mgr.y 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:12.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:12 smithi195 ceph-mon[25709]: Deploying daemon mgr.y on smithi149 2024-02-27T07:51:13.395 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:13 smithi149 systemd[1]: Stopping Ceph mgr.y for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:13.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:13 smithi149 podman[60125]: 2024-02-27 07:51:13.394754113 +0000 UTC m=+0.135317674 container died 0d1c71eebda2d74eca7c7243771dcb277a7cd430bafb6b578b2a7d2855833dcc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, ceph=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, RELEASE=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-02-27T07:51:14.007 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:13 smithi149 ceph-mon[29527]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:14.007 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:13 smithi149 ceph-mon[24796]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:14.008 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:13 smithi149 podman[60125]: 2024-02-27 07:51:13.780605228 +0000 UTC m=+0.521168787 container cleanup 0d1c71eebda2d74eca7c7243771dcb277a7cd430bafb6b578b2a7d2855833dcc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, architecture=x86_64, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, release=754, version=8, build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-27T07:51:14.008 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:13 smithi149 bash[60125]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y 2024-02-27T07:51:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:13 smithi195 ceph-mon[25709]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:14.331 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 podman[60139]: 2024-02-27 07:51:14.007099239 +0000 UTC m=+0.615467896 container remove 0d1c71eebda2d74eca7c7243771dcb277a7cd430bafb6b578b2a7d2855833dcc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, architecture=x86_64, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-27T07:51:14.332 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-27T07:51:14.332 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-27T07:51:14.332 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: Stopped Ceph mgr.y for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:14.332 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service: Consumed 53.186s CPU time. 2024-02-27T07:51:14.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: Starting Ceph mgr.y for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:14.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 podman[60274]: 2024-02-27 07:51:14.453082451 +0000 UTC m=+0.018699227 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:51:14.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 podman[60274]: 2024-02-27 07:51:14.558576309 +0000 UTC m=+0.124193119 container create c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-27T07:51:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 podman[60274]: 2024-02-27 07:51:14.785974258 +0000 UTC m=+0.351591029 container init c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-27T07:51:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 podman[60274]: 2024-02-27 07:51:14.788622696 +0000 UTC m=+0.354239468 container start c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-27T07:51:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 ceph-mgr[60293]: -- 172.21.15.149:0/865967593 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8581205a0 con 0x55a8580f4c00 2024-02-27T07:51:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 bash[60274]: c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 2024-02-27T07:51:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:14 smithi149 systemd[1]: Started Ceph mgr.y for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:15.458 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:15.354+0000 7f65fb83c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:51:15.458 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:15.456+0000 7f65fb83c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:51:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:15 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.160+0000 7f65fb83c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.253+0000 7f65fb83c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.255 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.256 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:15 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:16.508 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.401+0000 7f65fb83c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:51:16.508 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.507+0000 7f65fb83c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:51:16.835 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.592+0000 7f65fb83c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:51:17.101 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:17.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:17.102 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:16 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:16.881+0000 7f65fb83c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:51:17.102 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:17.006+0000 7f65fb83c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:17.102 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:17.099+0000 7f65fb83c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:51:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:16 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:17.395 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:17.205+0000 7f65fb83c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:51:17.396 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:17.394+0000 7f65fb83c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:51:18.100 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:51:17] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.33.4" 2024-02-27T07:51:18.101 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:17 smithi195 ceph-mon[25709]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:18.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-mon[24796]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:18.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:17 smithi149 ceph-mon[29527]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:18.598 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:18.435+0000 7f65fb83c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:51:18.598 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:18.597+0000 7f65fb83c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:51:18.888 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:51:19.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:18 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:18.950+0000 7f65fb83c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:51:19.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:19.043+0000 7f65fb83c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:51:19.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:19.260+0000 7f65fb83c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:19.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:19.459+0000 7f65fb83c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:51:20.048 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:19.768+0000 7f65fb83c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:20.048 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:20.046+0000 7f65fb83c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[29527]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[24796]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.309 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:20 smithi195 ceph-mon[25709]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:20 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:20 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:20 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:20 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:20.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:20.307+0000 7f65fb83c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:51:21.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:21 smithi149 ceph-mon[29527]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:21 smithi149 ceph-mon[24796]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:21.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:21 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:21.189+0000 7f65fb83c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:51:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:21 smithi195 ceph-mon[25709]: from='client.25132 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:22.075 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:21 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:21.812+0000 7f65fb83c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:51:22.075 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:21 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:21.917+0000 7f65fb83c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:51:22.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: Standby manager daemon y restarted 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: Standby manager daemon y started 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: Standby manager daemon y restarted 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: Standby manager daemon y started 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:22.156+0000 7f65fb83c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:22] ENGINE Bus STARTING 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: CherryPy Checker: 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: The Application mounted at '' has an empty config. 2024-02-27T07:51:22.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:51:22.429 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:22] ENGINE Serving on http://:::9283 2024-02-27T07:51:22.429 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:22 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:22] ENGINE Bus STARTED 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: Standby manager daemon y restarted 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: Standby manager daemon y started 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-27T07:51:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:22 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:23.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:23 smithi149 ceph-mon[24796]: mgrmap e26: x(active, since 82s), standbys: y 2024-02-27T07:51:23.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:23 smithi149 ceph-mon[29527]: mgrmap e26: x(active, since 82s), standbys: y 2024-02-27T07:51:23.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:23 smithi195 ceph-mon[25709]: mgrmap e26: x(active, since 82s), standbys: y 2024-02-27T07:51:24.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:24 smithi149 ceph-mon[29527]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:24.328 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:24 smithi149 ceph-mon[24796]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:24 smithi195 ceph-mon[25709]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[29527]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:51:26.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:25 smithi149 ceph-mon[24796]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: Reconfiguring iscsi.foo.smithi149.gjsccy (dependencies changed)... 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: Reconfiguring daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:51:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:25 smithi195 ceph-mon[25709]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:28.170 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:27 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:51:27] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.33.4" 2024-02-27T07:51:28.336 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:28 smithi149 ceph-mon[29527]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:28.336 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:28 smithi149 ceph-mon[24796]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:28.421 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-mon[25709]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:28.714 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:28.420+0000 7fd0391ed700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Creating ceph-iscsi config... 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:51:28.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return _run_code(code, main_globals, None, 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: exec(code, run_globals) 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-27T07:51:28.716 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2382, in _daemon_action 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return self.event_loop.get_result(coro, timeout) 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return future.result(timeout) 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return self.__get_result() 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: raise self._exception 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: config_blobs=daemon_spec.final_config, 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-27T07:51:28.717 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Creating ceph-iscsi config... 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Non-zero exit code 1 from systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T07:51:28.718 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: Traceback (most recent call last): 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: return _run_code(code, main_globals, None, 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: exec(code, run_globals) 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1070, in deploy_daemon 2024-02-27T07:51:28.719 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T07:51:28.720 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: RuntimeError: Failed command: systemctl restart ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T07:51:28.720 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T07:51:29.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:29.178+0000 7fd0a6b37700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-27T07:51:29.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ignoring --setuser ceph since I am not root 2024-02-27T07:51:29.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ignoring --setgroup ceph since I am not root 2024-02-27T07:51:29.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-mgr[46888]: -- 172.21.15.195:0/1953953191 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x565015c225a0 con 0x565015bf6800 2024-02-27T07:51:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-mon[25709]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-mon[25709]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:51:29.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[29527]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[29527]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE Bus STOPPING 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE Bus STOPPED 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE Bus STARTING 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE Serving on http://:::9283 2024-02-27T07:51:29.677 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:51:29] ENGINE Bus STARTED 2024-02-27T07:51:29.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:29.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[24796]: from='mgr.34877 172.21.15.195:0/565017440' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-27T07:51:29.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:29 smithi149 ceph-mon[24796]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:51:30.183 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:29.765+0000 7feddb822200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:51:30.183 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:29.863+0000 7feddb822200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: mgrmap e27: y(active, starting, since 0.74425s) 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:51:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: Manager daemon y is now available 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-mon[25709]: [27/Feb/2024:07:51:29] ENGINE Bus STARTING 2024-02-27T07:51:30.491 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-27T07:51:30.491 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: mgrmap e27: y(active, starting, since 0.74425s) 2024-02-27T07:51:30.491 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:51:30.491 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:51:30.492 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: Manager daemon y is now available 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[29527]: [27/Feb/2024:07:51:29] ENGINE Bus STARTING 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.34877 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: mgrmap e27: y(active, starting, since 0.74425s) 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:51:30.493 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: Manager daemon y is now available 2024-02-27T07:51:30.494 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:51:30.495 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:30 smithi149 ceph-mon[24796]: [27/Feb/2024:07:51:29] ENGINE Bus STARTING 2024-02-27T07:51:30.794 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:30.551+0000 7feddb822200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:51:30.794 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:30.647+0000 7feddb822200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:51:30.794 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:30.793+0000 7feddb822200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:51:31.197 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:30.899+0000 7feddb822200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:51:31.197 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:30.983+0000 7feddb822200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:51:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[29527]: [27/Feb/2024:07:51:29] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[29527]: [27/Feb/2024:07:51:30] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[29527]: [27/Feb/2024:07:51:30] ENGINE Bus STARTED 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[29527]: mgrmap e28: y(active, since 1.75899s) 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[29527]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[24796]: [27/Feb/2024:07:51:29] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[24796]: [27/Feb/2024:07:51:30] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[24796]: [27/Feb/2024:07:51:30] ENGINE Bus STARTED 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[24796]: mgrmap e28: y(active, since 1.75899s) 2024-02-27T07:51:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:31 smithi149 ceph-mon[24796]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:31.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:31.253+0000 7feddb822200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:51:31.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:31.370+0000 7feddb822200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-mon[25709]: [27/Feb/2024:07:51:29] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:51:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-mon[25709]: [27/Feb/2024:07:51:30] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:51:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-mon[25709]: [27/Feb/2024:07:51:30] ENGINE Bus STARTED 2024-02-27T07:51:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-mon[25709]: mgrmap e28: y(active, since 1.75899s) 2024-02-27T07:51:31.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-mon[25709]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:31.741 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:31.464+0000 7feddb822200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:51:31.741 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:31.560+0000 7feddb822200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:51:32.181 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:31 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:31.739+0000 7feddb822200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:32 smithi149 ceph-mon[24796]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:32 smithi149 ceph-mon[24796]: mgrmap e29: y(active, since 2s) 2024-02-27T07:51:32.213 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:32 smithi149 ceph-mon[29527]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:32.213 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:32 smithi149 ceph-mon[29527]: mgrmap e29: y(active, since 2s) 2024-02-27T07:51:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:32 smithi195 ceph-mon[25709]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:32 smithi195 ceph-mon[25709]: mgrmap e29: y(active, since 2s) 2024-02-27T07:51:33.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:32 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:32.766+0000 7feddb822200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:51:33.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:32 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:32.902+0000 7feddb822200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:51:33.548 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:33.244+0000 7feddb822200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:51:33.549 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:33.364+0000 7feddb822200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:51:33.814 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:33.566+0000 7feddb822200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:33.815 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:33.772+0000 7feddb822200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:51:34.070 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:51:34.379 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:34.096+0000 7feddb822200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:34.379 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:34.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: mgrmap e30: y(active, since 4s) 2024-02-27T07:51:34.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:34.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: mgrmap e30: y(active, since 4s) 2024-02-27T07:51:34.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: mgrmap e30: y(active, since 4s) 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:34 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:34.636 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:34.378+0000 7feddb822200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:34.636 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:34.634+0000 7feddb822200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:51:34.755 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:51:34.755 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (60s) 1s ago 5m 12.8M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:51:34.755 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (25s) 0s ago 5m 43.0M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:51:34.755 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 unknown 1s ago 4m - - 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283 running (104s) 0s ago 10m 73.6M - 19.0.0-1736-g635ee685 c4eb0759a73a 94caded06864 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (20s) 1s ago 11m 472M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (11m) 1s ago 12m 56.5M 2048M 17.2.0 e1d6a67b021e 649624a7b8b5 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (10m) 0s ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (10m) 1s ago 10m 48.9M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (69s) 1s ago 6m 8258k - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (41s) 0s ago 6m 8640k - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (9m) 1s ago 9m 52.1M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (9m) 1s ago 9m 52.8M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (9m) 1s ago 9m 47.5M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (8m) 1s ago 8m 51.5M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:51:34.756 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (8m) 0s ago 8m 53.1M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (8m) 0s ago 8m 49.2M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (7m) 0s ago 7m 49.0M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (7m) 0s ago 7m 50.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (43s) 0s ago 6m 52.9M - 2.33.4 514e6a882f6e 733eebc49216 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (5m) 1s ago 5m 70.5M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (5m) 0s ago 5m 70.7M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (4m) 1s ago 4m 72.4M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:51:34.757 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (4m) 0s ago 4m 72.4M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:51:35.210 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:51:35.210 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:51:35.210 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-27T07:51:35.210 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:35.210 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:51:35.211 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr" 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "2/23 daemons upgraded", 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "", 2024-02-27T07:51:35.558 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:51:35.559 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:51:35.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/1987414951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/1987414951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:35 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.953 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:35.477+0000 7feddb822200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/1987414951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:35.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:35.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:35 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:36.382 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:36.056+0000 7feddb822200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:51:36.382 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:36.157+0000 7feddb822200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:51:36.515 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:36.515 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:36.515 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/291618283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:36.515 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/291618283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: Standby manager daemon x started 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:36.516 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:36 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/291618283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/4141116543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:36.381+0000 7feddb822200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:36] ENGINE Bus STARTING 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: CherryPy Checker: 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: The Application mounted at '' has an empty config. 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:36] ENGINE Serving on http://:::9283 2024-02-27T07:51:36.715 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:36 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: [27/Feb/2024:07:51:36] ENGINE Bus STARTED 2024-02-27T07:51:37.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:37 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:51:37.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:37 smithi195 ceph-mon[25709]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-27T07:51:37.924 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:37 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:51:37.924 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:37 smithi149 ceph-mon[29527]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-27T07:51:37.924 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:37 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:51:37.924 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:37 smithi149 ceph-mon[24796]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-27T07:51:38.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:37 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[46884]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:51:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-27T07:51:38.954 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.955 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.956 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.956 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:38.956 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:38 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:51:38.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:38.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:38 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:39.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:39.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:39 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:40.085 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:51:40.085 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:40.086 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:39 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[29527]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[29527]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[29527]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2094171855' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[29527]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[24796]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[24796]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:51:41.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[24796]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:51:41.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[24796]: from='client.? 172.21.15.149:0/2094171855' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:51:41.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:40 smithi149 ceph-mon[24796]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:51:41.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:40 smithi195 ceph-mon[25709]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:51:41.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:40 smithi195 ceph-mon[25709]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-27T07:51:41.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:40 smithi195 ceph-mon[25709]: Reconfiguring daemon prometheus.a on smithi195 2024-02-27T07:51:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:40 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2094171855' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:51:41.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:40 smithi195 ceph-mon[25709]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:51:42.914 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 systemd[1]: Stopping Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:42.915 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 bash[53832]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:51:42.915 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-mon[25709]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:43.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:42 smithi149 ceph-mon[24796]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:43.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:42 smithi149 ceph-mon[29527]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:43.214 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.913Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[50513]: ts=2024-02-27T07:51:42.914Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-27T07:51:43.215 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:42 smithi195 podman[53845]: 2024-02-27 07:51:42.923996894 +0000 UTC m=+0.117177494 container died 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:43.556 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 podman[53845]: 2024-02-27 07:51:43.305119977 +0000 UTC m=+0.498300576 container cleanup 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:43.556 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 bash[53845]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a 2024-02-27T07:51:43.557 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 bash[53873]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T07:51:43.816 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 podman[53860]: 2024-02-27 07:51:43.556288114 +0000 UTC m=+0.633937231 container remove 733eebc49216b7efe4e314e2980b2fdf620ad07b8a1013c02cf2ea9131c98c37 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:43.816 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-27T07:51:43.816 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 systemd[1]: Stopped Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:43.816 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 systemd[1]: Starting Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:43.816 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 podman[53942]: 2024-02-27 07:51:43.694879677 +0000 UTC m=+0.018750431 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-27T07:51:44.078 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:43 smithi195 podman[53942]: 2024-02-27 07:51:43.815833392 +0000 UTC m=+0.139704178 container create 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:44.078 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 podman[53942]: 2024-02-27 07:51:44.051546055 +0000 UTC m=+0.375416796 container init 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:44.078 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 podman[53942]: 2024-02-27 07:51:44.053977103 +0000 UTC m=+0.377847844 container start 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T07:51:44.464 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-27T07:51:44.464 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:518 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi195 (none))" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.077Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.079Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.079Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.080Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.082Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.082Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.614µs 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.082Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 bash[53942]: 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 systemd[1]: Started Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:44.465 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.290Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-02-27T07:51:44.780 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:44.780 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:44.780 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:44.780 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:44.780 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:44.780 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.551Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-02-27T07:51:44.780 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.597Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-02-27T07:51:44.780 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.597Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.597Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.122µs wal_replay_duration=514.786576ms total_replay_duration=514.85282ms 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.601Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.601Z caller=main.go:947 level=info msg="TSDB started" 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.601Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.621Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.547712ms db_storage=894ns remote_storage=1.538µs web_handler=333ns query_engine=763ns scrape=607.064µs scrape_sd=98.215µs notify=10.259µs notify_sd=19.5µs rules=19.448619ms 2024-02-27T07:51:44.781 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:44.621Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-27T07:51:44.820 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:44.820 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:44 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:45.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:45.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:44 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:46.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:45 smithi149 ceph-mon[29527]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:46.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:45 smithi149 ceph-mon[24796]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:46.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:45 smithi195 ceph-mon[25709]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T07:51:46.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:46.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: Upgrade: Updating mgr.x 2024-02-27T07:51:46.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:46.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:46 smithi195 ceph-mon[25709]: Deploying daemon mgr.x on smithi195 2024-02-27T07:51:47.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: Upgrade: Updating mgr.x 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[24796]: Deploying daemon mgr.x on smithi195 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: Upgrade: Updating mgr.x 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:51:47.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:51:47.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:47.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:46 smithi149 ceph-mon[29527]: Deploying daemon mgr.x on smithi195 2024-02-27T07:51:47.959 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:47 smithi195 systemd[1]: Stopping Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:48.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:47 smithi195 podman[54285]: 2024-02-27 07:51:47.958243602 +0000 UTC m=+0.153123290 container died 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-27T07:51:48.604 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 podman[54285]: 2024-02-27 07:51:48.335521782 +0000 UTC m=+0.530401461 container cleanup 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T07:51:48.604 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 bash[54285]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:48 smithi195 ceph-mon[25709]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 podman[54299]: 2024-02-27 07:51:48.603806819 +0000 UTC m=+0.647340767 container remove 94caded06864e70889b52500bd1f16c06fc6b6d8ad05918f5d1bf92053101308 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 systemd[1]: Stopped Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service: Consumed 26.441s CPU time. 2024-02-27T07:51:48.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:48 smithi195 systemd[1]: Starting Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:51:49.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:48 smithi149 ceph-mon[29527]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:49.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:48 smithi149 ceph-mon[24796]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:51:49.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 podman[54433]: 2024-02-27 07:51:49.071303601 +0000 UTC m=+0.018351050 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:51:49.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 podman[54433]: 2024-02-27 07:51:49.181949917 +0000 UTC m=+0.128997348 container create 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-27T07:51:49.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 podman[54433]: 2024-02-27 07:51:49.417854539 +0000 UTC m=+0.364901970 container init 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-27T07:51:49.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 podman[54433]: 2024-02-27 07:51:49.420532892 +0000 UTC m=+0.367580322 container start 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-27T07:51:49.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 ceph-mgr[54453]: -- 172.21.15.195:0/862385260 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcc7a5a5a0 con 0x55bcc7a2f000 2024-02-27T07:51:49.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 bash[54433]: 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b 2024-02-27T07:51:49.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 systemd[1]: Started Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:51:50.449 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:49 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:49.976+0000 7f6e02bab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:51:50.449 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:50.083+0000 7f6e02bab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:51:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-mon[25709]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-02-27T07:51:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-mon[25709]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:50.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[24796]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[24796]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[29527]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:50.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:50 smithi149 ceph-mon[29527]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:51:51.020 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:50.764+0000 7f6e02bab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:51:51.020 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:50.874+0000 7f6e02bab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:51:51.349 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.019+0000 7f6e02bab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:51:51.350 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.116+0000 7f6e02bab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:51:51.350 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.203+0000 7f6e02bab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:51:51.698 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.477+0000 7f6e02bab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:51:51.698 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.595+0000 7f6e02bab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:51.698 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.696+0000 7f6e02bab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:51:51.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.810+0000 7f6e02bab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:51:52.286 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:51 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:51.997+0000 7f6e02bab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:51:52.857 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:52 smithi195 ceph-mon[25709]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-27T07:51:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:52 smithi149 ceph-mon[24796]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-27T07:51:52.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:52 smithi149 ceph-mon[29527]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-27T07:51:53.161 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:53.016+0000 7f6e02bab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:51:53.161 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:53.159+0000 7f6e02bab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:51:53.818 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:53.509+0000 7f6e02bab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:51:53.818 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:53.603+0000 7f6e02bab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:51:53.819 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:53 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:53.817+0000 7f6e02bab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:54.214 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:54.020+0000 7f6e02bab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:51:54.606 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:54.328+0000 7f6e02bab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:51:54.861 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:54.605+0000 7f6e02bab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:51:54.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-mon[25709]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:54.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:54.862 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:51:54] "GET /metrics HTTP/1.1" 200 38247 "" "Prometheus/2.33.4" 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[24796]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[29527]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:55.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:55.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:54 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:55.203 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:54.860+0000 7f6e02bab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:51:55.891 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:55 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:55.732+0000 7f6e02bab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:51:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:55 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:55 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:55 smithi195 ceph-mon[25709]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:56.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[24796]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:55 smithi149 ceph-mon[29527]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:51:56.652 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:56.324+0000 7f6e02bab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:51:56.652 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:56.426+0000 7f6e02bab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:56.650+0000 7f6e02bab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:51:56] ENGINE Bus STARTING 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: CherryPy Checker: 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: The Application mounted at '' has an empty config. 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:51:56] ENGINE Serving on http://:::9283 2024-02-27T07:51:56.956 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:51:56 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:51:56] ENGINE Bus STARTED 2024-02-27T07:51:57.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: Standby manager daemon x restarted 2024-02-27T07:51:57.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:51:57.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:57.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:57.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:57.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:57 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: Standby manager daemon x restarted 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: Standby manager daemon x started 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[24796]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: Standby manager daemon x restarted 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:51:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:51:57.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:51:57.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:51:57.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:57 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/3136444741' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:51:58.360 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:58 smithi195 ceph-mon[25709]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-27T07:51:58.361 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:58 smithi195 ceph-mon[25709]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:58.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:58 smithi149 ceph-mon[24796]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-27T07:51:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:58 smithi149 ceph-mon[24796]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:58 smithi149 ceph-mon[29527]: mgrmap e32: y(active, since 28s), standbys: x 2024-02-27T07:51:58.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:58 smithi149 ceph-mon[29527]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:51:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:51:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:51:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:51:59.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: Upgrade: Setting container_image for all mgr 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-27T07:51:59.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:51:59.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-27T07:51:59.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: Upgrade: It appears safe to stop mon.a 2024-02-27T07:51:59.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:51:59 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:59.724 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: Upgrade: Setting container_image for all mgr 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.725 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: Upgrade: It appears safe to stop mon.a 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-02-27T07:51:59.726 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: Upgrade: Setting container_image for all mgr 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-27T07:51:59.727 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: Upgrade: It appears safe to stop mon.a 2024-02-27T07:51:59.728 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:51:59 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:00.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[29527]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:00.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[29527]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[24796]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[24796]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:00 smithi149 ceph-mon[24796]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:00 smithi195 ceph-mon[25709]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:00 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='mgr.y' 2024-02-27T07:52:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:00 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:00 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:00 smithi195 ceph-mon[25709]: from='mgr.15330 172.21.15.149:0/2880196032' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[29527]: Upgrade: Updating mon.a 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[29527]: Deploying daemon mon.a on smithi149 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[29527]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[24796]: Upgrade: Updating mon.a 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[24796]: Deploying daemon mon.a on smithi149 2024-02-27T07:52:01.663 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:01 smithi149 ceph-mon[24796]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:01.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:01 smithi195 ceph-mon[25709]: Upgrade: Updating mon.a 2024-02-27T07:52:01.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:01 smithi195 ceph-mon[25709]: Deploying daemon mon.a on smithi149 2024-02-27T07:52:01.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:01 smithi195 ceph-mon[25709]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:02.460 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 systemd[1]: Stopping Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:02.785 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[24792]: 2024-02-27T07:52:02.458+0000 7f617330e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:52:02.785 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[24792]: 2024-02-27T07:52:02.458+0000 7f617330e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-27T07:52:02.785 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 podman[64425]: 2024-02-27 07:52:02.509697923 +0000 UTC m=+0.169536059 container died 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, release=754, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-02-27T07:52:03.143 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 podman[64425]: 2024-02-27 07:52:02.875362223 +0000 UTC m=+0.535200358 container cleanup 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, distribution-scope=public, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, vcs-type=git) 2024-02-27T07:52:03.144 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:02 smithi149 bash[64425]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a 2024-02-27T07:52:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 podman[64439]: 2024-02-27 07:52:03.143246213 +0000 UTC m=+0.634030824 container remove 649624a7b8b5ac28c5825cc440012a68ec51dcb4ff077715a58f8a23e43f0136 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-type=git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux ) 2024-02-27T07:52:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-27T07:52:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 systemd[1]: Stopped Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:03.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service: Consumed 13.536s CPU time. 2024-02-27T07:52:03.843 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 systemd[1]: Starting Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:03.843 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 podman[64570]: 2024-02-27 07:52:03.624339615 +0000 UTC m=+0.021492339 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:52:03.843 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 podman[64570]: 2024-02-27 07:52:03.735741293 +0000 UTC m=+0.132894053 container create 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:52:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 podman[64570]: 2024-02-27 07:52:03.971745203 +0000 UTC m=+0.368897904 container init 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5) 2024-02-27T07:52:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 podman[64570]: 2024-02-27 07:52:03.974862594 +0000 UTC m=+0.372015296 container start 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-02-27T07:52:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 ceph-mon[64590]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T07:52:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 ceph-mon[64590]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev), process ceph-mon, pid 2 2024-02-27T07:52:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:03 smithi149 ceph-mon[64590]: pidfile_write: ignore empty --pid-file 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: load: jerasure load: lrc 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: RocksDB version: 7.9.2 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Git sha 0 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Compile date 2024-02-26 21:43:15 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: DB SUMMARY 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: DB Session ID: VLUFQMASAPXK6H6RQP0Z 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: CURRENT file: CURRENT 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3221469 ; 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.error_if_exists: 0 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.create_if_missing: 0 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.paranoid_checks: 1 2024-02-27T07:52:04.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.env: 0x556a1edefb20 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.fs: PosixFileSystem 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.info_log: 0x556a1fa118e0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.statistics: (nil) 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.use_fsync: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_log_file_size: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_fallocate: 1 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T07:52:04.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.use_direct_reads: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.db_log_dir: 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.wal_dir: 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.write_buffer_manager: 0x556a1f9e4dc0 2024-02-27T07:52:04.180 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.unordered_write: 0 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.row_cache: None 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.wal_filter: None 2024-02-27T07:52:04.181 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.two_write_queues: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.wal_compression: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.atomic_flush: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.log_readahead_size: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T07:52:04.183 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_background_jobs: 2 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_background_compactions: -1 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_subcompactions: 1 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_open_files: -1 2024-02-27T07:52:04.184 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_background_flushes: -1 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Compression algorithms supported: 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kZSTD supported: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kXpressCompression supported: 0 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kZlibCompression supported: 1 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kSnappyCompression supported: 1 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kLZ4Compression supported: 1 2024-02-27T07:52:04.185 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: kBZip2Compression supported: 0 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.merge_operator: 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_filter: None 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556a1fa119e0) 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-02-27T07:52:04.186 INFO:journalctl@ceph.mon.a.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_type: 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: data_block_index_type: 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_shortening: 1 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: checksum: 4 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: no_block_cache: 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache: 0x556a1fa15090 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_options: 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: capacity : 536870912 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: num_shard_bits : 4 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: strict_capacity_limit : 0 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-02-27T07:52:04.187 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_cache_compressed: (nil) 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: persistent_cache: (nil) 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_size: 4096 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_size_deviation: 10 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_restart_interval: 16 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: index_block_restart_interval: 1 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: metadata_block_size: 4096 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: partition_filters: 0 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: use_delta_encoding: 1 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: filter_policy: bloomfilter 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: whole_key_filtering: 1 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: verify_compression: 0 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: format_version: 5 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: enable_index_compression: 1 2024-02-27T07:52:04.188 INFO:journalctl@ceph.mon.a.smithi149.stdout: block_align: 0 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout: max_auto_readahead_size: 262144 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout: prepopulate_block_cache: 0 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout: initial_auto_readahead_size: 8192 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression: NoCompression 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.num_levels: 7 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T07:52:04.189 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:04.190 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T07:52:04.191 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.arena_block_size: 1048576 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T07:52:04.192 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.table_properties_collectors: 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.inplace_update_support: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.bloom_locality: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.max_successive_merges: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T07:52:04.193 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.ttl: 2592000 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enable_blob_files: false 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.min_blob_size: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.blob_file_starting_level: 0 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-27T07:52:04.194 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12184, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b07cc65d-8bef-41c0-9129-ce2727e30d09 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020324004871, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020324015835, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2956999, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12185, "largest_seqno": 13496, "table_properties": {"data_size": 2950639, "index_size": 3869, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 13829, "raw_average_key_size": 23, "raw_value_size": 2937929, "raw_average_value_size": 5065, "num_data_blocks": 177, "num_entries": 580, "num_filter_entries": 580, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709020324, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b07cc65d-8bef-41c0-9129-ce2727e30d09", "db_session_id": "VLUFQMASAPXK6H6RQP0Z", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020324015948, "job": 1, "event": "recovery_finished"} 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556a1fb06000 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: DB pointer 0x556a1faf0000 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** DB Stats ** 2024-02-27T07:52:04.195 INFO:journalctl@ceph.mon.a.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: L0 1/0 2.82 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: L6 1/0 7.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Sum 2/0 10.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T07:52:04.196 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 640.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Cumulative compaction: 0.00 GB write, 184.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:04.197 INFO:journalctl@ceph.mon.a.smithi149.stdout: Interval compaction: 0.00 GB write, 184.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: Block cache BinnedLRUCache@0x556a1fa15090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: starting mon.a rank 0 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???) e3 preinit fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).mds e1 new map 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).mds e1 print_map 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: e1 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: legacy client fscid: -1 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: 2024-02-27T07:52:04.198 INFO:journalctl@ceph.mon.a.smithi149.stdout: No filesystems configured 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-mon[64590]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 bash[64570]: 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 2024-02-27T07:52:04.199 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:04 smithi149 systemd[1]: Started Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:05.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:04] "GET /metrics HTTP/1.1" 200 38242 "" "Prometheus/2.33.4" 2024-02-27T07:52:06.319 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:52:06.979 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (92s) 33s ago 6m 12.8M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (57s) 13s ago 6m 43.1M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 unknown 33s ago 5m - - 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (17s) 13s ago 10m 66.4M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (52s) 33s ago 12m 472M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 starting - - - 2048M 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (11m) 13s ago 11m 47.0M 2048M 17.2.0 e1d6a67b021e 89f749c4a6e0 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (11m) 33s ago 11m 48.9M 2048M 17.2.0 e1d6a67b021e 4d54cda6d15d 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (101s) 33s ago 7m 8258k - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (73s) 13s ago 7m 15.0M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:52:06.980 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (10m) 33s ago 10m 52.1M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (10m) 33s ago 10m 52.8M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (9m) 33s ago 9m 47.5M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (9m) 33s ago 9m 51.5M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (8m) 13s ago 8m 53.2M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (8m) 13s ago 8m 49.3M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (8m) 13s ago 8m 48.4M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (7m) 13s ago 7m 51.3M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (23s) 13s ago 7m 46.4M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (5m) 33s ago 5m 70.5M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (5m) 13s ago 5m 71.3M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (5m) 33s ago 5m 72.4M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:52:06.981 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (5m) 13s ago 5m 73.0M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:52:07.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ignoring --setuser ceph since I am not root 2024-02-27T07:52:07.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ignoring --setgroup ceph since I am not root 2024-02-27T07:52:07.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mgr[60293]: -- 172.21.15.149:0/3280860129 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb0c1025a0 con 0x55fb0c0d7000 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: mon.a calling monitor election 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: fsmap 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-27T07:52:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: from='mgr.15330 ' entity='' 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[64590]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: mon.a calling monitor election 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: fsmap 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: from='mgr.15330 ' entity='' 2024-02-27T07:52:07.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-mon[29527]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-27T07:52:07.429 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:52:07.429 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:52:07.429 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-27T07:52:07.429 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "mds": {}, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:52:07.430 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:52:07.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: ignoring --setuser ceph since I am not root 2024-02-27T07:52:07.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: ignoring --setgroup ceph since I am not root 2024-02-27T07:52:07.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mgr[54453]: -- 172.21.15.195:0/1638884409 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d731d2a5a0 con 0x55d731cfe800 2024-02-27T07:52:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: mon.a calling monitor election 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: fsmap 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: mgrmap e32: y(active, since 38s), standbys: x 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: from='mgr.15330 ' entity='' 2024-02-27T07:52:07.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-mon[25709]: mgrmap e33: y(active, since 38s), standbys: x 2024-02-27T07:52:07.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:07.662+0000 7fd748cb7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:52:07.765 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:07.666+0000 7f6470db7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-27T07:52:08.118 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:07 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:07.757+0000 7fd748cb7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:52:08.118 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:07.763+0000 7f6470db7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-27T07:52:08.407 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3019976856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:08.407 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-mon[64590]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-27T07:52:08.407 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/3019976856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:08.407 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-mon[29527]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-27T07:52:08.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/3019976856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:08.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-mon[25709]: mgrmap e34: y(active, since 39s), standbys: x 2024-02-27T07:52:08.659 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:08.429+0000 7f6470db7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:52:08.659 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:08.524+0000 7f6470db7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:52:08.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:08.405+0000 7fd748cb7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-27T07:52:08.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:08.501+0000 7fd748cb7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-27T07:52:08.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:08.645+0000 7fd748cb7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:52:08.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:08.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:08.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:08.665+0000 7f6470db7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-27T07:52:08.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:08.761+0000 7f6470db7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:52:08.965 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:08.843+0000 7f6470db7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:52:09.094 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:08.742+0000 7fd748cb7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-27T07:52:09.094 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:08 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:08.825+0000 7fd748cb7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-27T07:52:09.094 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:09.093+0000 7fd748cb7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:52:09.388 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:09.204+0000 7fd748cb7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:52:09.388 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:09.291+0000 7fd748cb7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:52:09.388 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:09.386+0000 7fd748cb7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:52:09.392 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:09.105+0000 7f6470db7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-27T07:52:09.392 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:09.213+0000 7f6470db7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-27T07:52:09.392 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:09.298+0000 7f6470db7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-27T07:52:09.392 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:09.390+0000 7f6470db7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-27T07:52:09.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:09.567+0000 7fd748cb7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:52:09.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:09.568+0000 7f6470db7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-27T07:52:10.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:10 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:10.534+0000 7fd748cb7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:52:10.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:10 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:10.665+0000 7fd748cb7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:52:10.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:10.529+0000 7f6470db7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-27T07:52:10.964 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:10.658+0000 7f6470db7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-27T07:52:11.283 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:10.993+0000 7f6470db7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:52:11.283 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:11.085+0000 7f6470db7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:52:11.301 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:11.007+0000 7fd748cb7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-27T07:52:11.301 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:11.099+0000 7fd748cb7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-27T07:52:11.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:11.300+0000 7fd748cb7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:52:11.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:11.487+0000 7fd748cb7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:52:11.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:11.282+0000 7f6470db7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:52:11.714 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:11.467+0000 7f6470db7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-27T07:52:12.013 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:11.753+0000 7f6470db7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:52:12.043 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:11.775+0000 7fd748cb7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-27T07:52:12.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:12.042+0000 7fd748cb7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:52:12.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:12 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:12.283+0000 7fd748cb7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:52:12.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:12.013+0000 7f6470db7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-27T07:52:12.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:12.250+0000 7f6470db7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-27T07:52:13.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:13.101+0000 7fd748cb7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:52:13.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:13.055+0000 7f6470db7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-27T07:52:13.953 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:13.629+0000 7f6470db7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:52:13.953 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:13.734+0000 7f6470db7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:52:14.020 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:13.689+0000 7fd748cb7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-27T07:52:14.020 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:13 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:13.795+0000 7fd748cb7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-27T07:52:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: Standby manager daemon x restarted 2024-02-27T07:52:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: Standby manager daemon x started 2024-02-27T07:52:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:52:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:52:14.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-mon[25709]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:13.953+0000 7f6470db7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:52:13] ENGINE Bus STARTING 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: CherryPy Checker: 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: The Application mounted at '' has an empty config. 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:52:14] ENGINE Serving on http://:::9283 2024-02-27T07:52:14.215 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 07:52:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x[54449]: [27/Feb/2024:07:52:14] ENGINE Bus STARTED 2024-02-27T07:52:14.272 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: Standby manager daemon x restarted 2024-02-27T07:52:14.272 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: Standby manager daemon x started 2024-02-27T07:52:14.272 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:52:14.272 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[64590]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: Standby manager daemon x restarted 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: Standby manager daemon x started 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-mon[29527]: from='mgr.? 172.21.15.195:0/2988547701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T07:52:14.273 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:14.029+0000 7fd748cb7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:52:14] ENGINE Bus STARTING 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: CherryPy Checker: 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: The Application mounted at '' has an empty config. 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:52:14] ENGINE Serving on http://:::9283 2024-02-27T07:52:14.274 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:07:52:14] ENGINE Bus STARTED 2024-02-27T07:52:15.049 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr" 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "2/23 daemons upgraded", 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "", 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:52:15.050 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:14] "GET /metrics HTTP/1.1" 200 35459 "" "Prometheus/2.33.4" 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:15.051+0000 7fd6feeab700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: Active manager daemon y restarted 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: Activating manager daemon y 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: mgrmap e36: y(active, starting, since 0.00917632s), standbys: x 2024-02-27T07:52:15.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:52:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: Manager daemon y is now available 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: Active manager daemon y restarted 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: Activating manager daemon y 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: mgrmap e36: y(active, starting, since 0.00917632s), standbys: x 2024-02-27T07:52:15.179 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:52:15.180 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: Manager daemon y is now available 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:52:15.181 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:15 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:52:15.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: mgrmap e35: y(active, since 45s), standbys: x 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: Active manager daemon y restarted 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: Activating manager daemon y 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: mgrmap e36: y(active, starting, since 0.00917632s), standbys: x 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:52:15.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: Manager daemon y is now available 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-27T07:52:15.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:15 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-27T07:52:15.462 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-27T07:52:15.462 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T07:52:15.462 INFO:teuthology.orchestra.run.smithi149.stdout: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: [27/Feb/2024:07:52:14] ENGINE Bus STARTING 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: [27/Feb/2024:07:52:14] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: [27/Feb/2024:07:52:14] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: [27/Feb/2024:07:52:14] ENGINE Bus STARTED 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: from='client.44118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:16.204 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: mgrmap e37: y(active, since 1.01356s), standbys: x 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[29527]: from='client.? 172.21.15.149:0/2510198163' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: [27/Feb/2024:07:52:14] ENGINE Bus STARTING 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: [27/Feb/2024:07:52:14] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: [27/Feb/2024:07:52:14] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: [27/Feb/2024:07:52:14] ENGINE Bus STARTED 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: from='client.44118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: mgrmap e37: y(active, since 1.01356s), standbys: x 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:16.205 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:16 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2510198163' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: [27/Feb/2024:07:52:14] ENGINE Bus STARTING 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: [27/Feb/2024:07:52:14] ENGINE Serving on http://172.21.15.149:8765 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: [27/Feb/2024:07:52:14] ENGINE Serving on https://172.21.15.149:7150 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: [27/Feb/2024:07:52:14] ENGINE Bus STARTED 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: from='client.44118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: mgrmap e37: y(active, since 1.01356s), standbys: x 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:16 smithi195 ceph-mon[25709]: from='client.? 172.21.15.149:0/2510198163' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:17.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:17 smithi195 ceph-mon[25709]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-27T07:52:17.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:17 smithi195 ceph-mon[25709]: Cluster is now healthy 2024-02-27T07:52:17.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:17 smithi195 ceph-mon[25709]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:17.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[29527]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-27T07:52:17.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[29527]: Cluster is now healthy 2024-02-27T07:52:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[29527]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[64590]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-27T07:52:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:52:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:17 smithi149 ceph-mon[64590]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:18.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:18 smithi195 ceph-mon[25709]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-27T07:52:18.321 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:18 smithi149 ceph-mon[29527]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-27T07:52:18.321 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:18 smithi149 ceph-mon[64590]: mgrmap e38: y(active, since 3s), standbys: x 2024-02-27T07:52:18.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:19.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[64590]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:19.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[29527]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:19 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:19 smithi195 ceph-mon[25709]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:52:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:19 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:19 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:19 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:19 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: mgrmap e39: y(active, since 5s), standbys: x 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:52:20.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: mgrmap e39: y(active, since 5s), standbys: x 2024-02-27T07:52:20.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: mgrmap e39: y(active, since 5s), standbys: x 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:20.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:20 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-02-27T07:52:21.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:21 smithi195 ceph-mon[25709]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:52:21.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:21 smithi149 ceph-mon[29527]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:52:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:21 smithi149 ceph-mon[64590]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-27T07:52:23.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:23 smithi149 ceph-mon[29527]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:52:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:23 smithi149 ceph-mon[64590]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:52:23.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:23 smithi195 ceph-mon[25709]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[29527]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-mon[64590]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:52:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.conf 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.conf 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: Updating smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T07:52:24.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:24 smithi195 ceph-mon[25709]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-27T07:52:25.118 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:24] "GET /metrics HTTP/1.1" 200 35459 "" "Prometheus/2.33.4" 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: Upgrade: It appears safe to stop mon.c 2024-02-27T07:52:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[29527]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: Upgrade: It appears safe to stop mon.c 2024-02-27T07:52:25.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:25 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:25.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:25.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:25.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:25.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-27T07:52:25.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: Upgrade: It appears safe to stop mon.c 2024-02-27T07:52:25.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:25 smithi195 ceph-mon[25709]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:27.113 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: Upgrade: Updating mon.c 2024-02-27T07:52:27.113 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:27.113 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: Deploying daemon mon.c on smithi149 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[64590]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: Upgrade: Updating mon.c 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: Deploying daemon mon.c on smithi149 2024-02-27T07:52:27.114 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:26 smithi149 ceph-mon[29527]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: Upgrade: Updating mon.c 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: Deploying daemon mon.c on smithi149 2024-02-27T07:52:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:26 smithi195 ceph-mon[25709]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-27T07:52:27.918 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:27 smithi149 systemd[1]: Stopping Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:28.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c[29523]: 2024-02-27T07:52:27.916+0000 7f89c2569700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:52:28.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:27 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c[29523]: 2024-02-27T07:52:27.916+0000 7f89c2569700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-27T07:52:28.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 podman[66990]: 2024-02-27 07:52:28.050342576 +0000 UTC m=+0.245132010 container died 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, RELEASE=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., release=754, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-02-27T07:52:28.660 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 podman[66990]: 2024-02-27 07:52:28.450199676 +0000 UTC m=+0.644989109 container cleanup 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, release=754, io.openshift.expose-services=, RELEASE=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git) 2024-02-27T07:52:28.660 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 bash[66990]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c 2024-02-27T07:52:28.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 podman[67004]: 2024-02-27 07:52:28.659741149 +0000 UTC m=+0.612889166 container remove 4d54cda6d15db6535657905ec143b91f96f9824c9815c5db984f8960ddcb2fb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, distribution-scope=public, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, ceph=True) 2024-02-27T07:52:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-27T07:52:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 systemd[1]: Stopped Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:28.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.c.service: Consumed 7.807s CPU time. 2024-02-27T07:52:28.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:29.363 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:28 smithi149 systemd[1]: Starting Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:29.363 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 podman[67138]: 2024-02-27 07:52:29.114746767 +0000 UTC m=+0.017854870 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:52:29.364 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 podman[67138]: 2024-02-27 07:52:29.236226751 +0000 UTC m=+0.139334891 container create b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:52:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 podman[67138]: 2024-02-27 07:52:29.45536098 +0000 UTC m=+0.358469080 container init b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True) 2024-02-27T07:52:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 podman[67138]: 2024-02-27 07:52:29.458597299 +0000 UTC m=+0.361705397 container start b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-27T07:52:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T07:52:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev), process ceph-mon, pid 2 2024-02-27T07:52:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: pidfile_write: ignore empty --pid-file 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: load: jerasure load: lrc 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: RocksDB version: 7.9.2 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Git sha 0 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Compile date 2024-02-26 21:43:15 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: DB SUMMARY 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: DB Session ID: 3MFSP6HXFDKKG8ZEXW4T 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: CURRENT file: CURRENT 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-02-27T07:52:29.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 941976 ; 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.error_if_exists: 0 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.create_if_missing: 0 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.paranoid_checks: 1 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.env: 0x56293a911b20 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.fs: PosixFileSystem 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.info_log: 0x56293b2018e0 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.statistics: (nil) 2024-02-27T07:52:29.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.use_fsync: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_log_file_size: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_fallocate: 1 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.use_direct_reads: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.db_log_dir: 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.wal_dir: 2024-02-27T07:52:29.680 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.write_buffer_manager: 0x56293b1d4dc0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T07:52:29.681 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.unordered_write: 0 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.row_cache: None 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.wal_filter: None 2024-02-27T07:52:29.682 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T07:52:29.683 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T07:52:29.683 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.two_write_queues: 0 2024-02-27T07:52:29.683 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.wal_compression: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.atomic_flush: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.log_readahead_size: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T07:52:29.684 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_background_jobs: 2 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_background_compactions: -1 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_subcompactions: 1 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_open_files: -1 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T07:52:29.685 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_background_flushes: -1 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Compression algorithms supported: 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kZSTD supported: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kXpressCompression supported: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kZlibCompression supported: 1 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kSnappyCompression supported: 1 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kLZ4Compression supported: 1 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: kBZip2Compression supported: 0 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-02-27T07:52:29.686 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.merge_operator: 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_filter: None 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56293b2019e0) 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_type: 0 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: data_block_index_type: 0 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_shortening: 1 2024-02-27T07:52:29.687 INFO:journalctl@ceph.mon.c.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: checksum: 4 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: no_block_cache: 0 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache: 0x56293b205090 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_options: 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: capacity : 536870912 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: num_shard_bits : 4 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: strict_capacity_limit : 0 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_cache_compressed: (nil) 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: persistent_cache: (nil) 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_size: 4096 2024-02-27T07:52:29.688 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_size_deviation: 10 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_restart_interval: 16 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: index_block_restart_interval: 1 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: metadata_block_size: 4096 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: partition_filters: 0 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: use_delta_encoding: 1 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: filter_policy: bloomfilter 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: whole_key_filtering: 1 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: verify_compression: 0 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: format_version: 5 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: enable_index_compression: 1 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: block_align: 0 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: max_auto_readahead_size: 262144 2024-02-27T07:52:29.689 INFO:journalctl@ceph.mon.c.smithi149.stdout: prepopulate_block_cache: 0 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout: initial_auto_readahead_size: 8192 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression: NoCompression 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.num_levels: 7 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T07:52:29.690 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:29.691 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T07:52:29.692 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.arena_block_size: 1048576 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T07:52:29.693 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.table_properties_collectors: 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.inplace_update_support: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.bloom_locality: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.max_successive_merges: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T07:52:29.694 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.ttl: 2592000 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enable_blob_files: false 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.min_blob_size: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.blob_file_starting_level: 0 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-27T07:52:29.695 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14802, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 050f5c90-7158-474c-a6be-0b79ce3fccfe 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020349487997, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020349491198, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 585031, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14807, "largest_seqno": 15040, "table_properties": {"data_size": 582843, "index_size": 856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2783, "raw_average_key_size": 24, "raw_value_size": 580109, "raw_average_value_size": 5133, "num_data_blocks": 38, "num_entries": 113, "num_filter_entries": 113, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709020349, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "050f5c90-7158-474c-a6be-0b79ce3fccfe", "db_session_id": "3MFSP6HXFDKKG8ZEXW4T", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020349491287, "job": 1, "event": "recovery_finished"} 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56293b2f6000 2024-02-27T07:52:29.696 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: DB pointer 0x56293b2e0000 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** DB Stats ** 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-27T07:52:29.697 INFO:journalctl@ceph.mon.c.smithi149.stdout: L0 1/0 571.32 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 329.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: L6 1/0 11.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Sum 2/0 11.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 329.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 329.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 329.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:52:29.698 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: Cumulative compaction: 0.00 GB write, 105.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: Interval compaction: 0.00 GB write, 105.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: Block cache BinnedLRUCache@0x56293b205090#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.95 KB,0.000181794%) Misc(1,0.00 KB,0%) 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: starting mon.c rank 1 at public addrs [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] at bind addrs [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???) e3 preinit fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).mds e1 new map 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).mds e1 print_map 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: e1 2024-02-27T07:52:29.699 INFO:journalctl@ceph.mon.c.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout: legacy client fscid: -1 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout: 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout: No filesystems configured 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 ceph-mon[67157]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 bash[67138]: b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 2024-02-27T07:52:29.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:29 smithi149 systemd[1]: Started Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: mon.c calling monitor election 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: mon.a calling monitor election 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: fsmap 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: mgrmap e39: y(active, since 15s), standbys: x 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: overall HEALTH_OK 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[67157]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: mon.c calling monitor election 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: mon.a calling monitor election 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: fsmap 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: mgrmap e39: y(active, since 15s), standbys: x 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: overall HEALTH_OK 2024-02-27T07:52:30.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:30.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:30 smithi149 ceph-mon[64590]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: mon.c calling monitor election 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: mon.a calling monitor election 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: monmap e3: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: fsmap 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:30.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: mgrmap e39: y(active, since 15s), standbys: x 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: overall HEALTH_OK 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:30.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:30 smithi195 ceph-mon[25709]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-27T07:52:33.372 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:33 smithi149 ceph-mon[67157]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:33.372 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:33 smithi149 ceph-mon[64590]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:33 smithi195 ceph-mon[25709]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:34.809 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:34.498+0000 7fd6feeab700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-27T07:52:35.068 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.068 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.069 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[64590]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:35.069 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:34] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.33.4" 2024-02-27T07:52:35.069 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.069 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.069 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:34 smithi149 ceph-mon[67157]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:35.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:34 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:34 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:35.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:34 smithi195 ceph-mon[25709]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:36.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:35 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:36.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:35 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:35 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:36.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:35 smithi195 ceph-mon[25709]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:37.115 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:36 smithi195 ceph-mon[25709]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:37.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:36 smithi149 ceph-mon[64590]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:37.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:36 smithi149 ceph-mon[67157]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:38.952 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:39.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:39 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-27T07:52:39.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-27T07:52:39.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T07:52:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-27T07:52:40.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:40 smithi195 ceph-mon[25709]: Upgrade: It appears safe to stop mon.b 2024-02-27T07:52:40.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:40 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:40.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:40 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:40.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:40 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:40.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:40 smithi195 ceph-mon[25709]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:40.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[67157]: Upgrade: It appears safe to stop mon.b 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[64590]: Upgrade: It appears safe to stop mon.b 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:40 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:41.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[67157]: Upgrade: Updating mon.b 2024-02-27T07:52:41.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[67157]: Deploying daemon mon.b on smithi195 2024-02-27T07:52:41.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[67157]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:41.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[64590]: Upgrade: Updating mon.b 2024-02-27T07:52:41.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[64590]: Deploying daemon mon.b on smithi195 2024-02-27T07:52:41.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:41 smithi149 ceph-mon[64590]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:41 smithi195 ceph-mon[25709]: Upgrade: Updating mon.b 2024-02-27T07:52:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:41 smithi195 ceph-mon[25709]: Deploying daemon mon.b on smithi195 2024-02-27T07:52:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:41 smithi195 ceph-mon[25709]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:42.005 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:41 smithi195 systemd[1]: Stopping Ceph mon.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b[25702]: 2024-02-27T07:52:42.002+0000 7f019b73f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:52:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b[25702]: 2024-02-27T07:52:42.002+0000 7f019b73f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-27T07:52:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 podman[57930]: 2024-02-27 07:52:42.228017387 +0000 UTC m=+0.344599745 container died 89f749c4a6e07ce5f254bba3ed023c747ae7bcfff6e6ce7f1b142b5599c5183e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, version=8, io.openshift.expose-services=, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-02-27T07:52:42.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 podman[57930]: 2024-02-27 07:52:42.620063169 +0000 UTC m=+0.736645528 container cleanup 89f749c4a6e07ce5f254bba3ed023c747ae7bcfff6e6ce7f1b142b5599c5183e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-27T07:52:42.872 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 bash[57930]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b 2024-02-27T07:52:43.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:42 smithi195 podman[57944]: 2024-02-27 07:52:42.871475145 +0000 UTC m=+0.636866874 container remove 89f749c4a6e07ce5f254bba3ed023c747ae7bcfff6e6ce7f1b142b5599c5183e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-type=git, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.buildah.version=1.19.8, architecture=x86_64, vendor=Red Hat, Inc., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-27T07:52:43.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-27T07:52:43.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 systemd[1]: Stopped Ceph mon.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:43.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.b.service: Consumed 9.129s CPU time. 2024-02-27T07:52:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 systemd[1]: Starting Ceph mon.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:52:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 podman[58075]: 2024-02-27 07:52:43.361123877 +0000 UTC m=+0.018822670 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:52:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 podman[58075]: 2024-02-27 07:52:43.497898211 +0000 UTC m=+0.155597067 container create 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:52:43.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 podman[58075]: 2024-02-27 07:52:43.742076233 +0000 UTC m=+0.399775026 container init 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:52:43.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 podman[58075]: 2024-02-27 07:52:43.745183629 +0000 UTC m=+0.402882426 container start 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T07:52:44.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T07:52:44.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev), process ceph-mon, pid 2 2024-02-27T07:52:44.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: pidfile_write: ignore empty --pid-file 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: load: jerasure load: lrc 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: RocksDB version: 7.9.2 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Git sha 0 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Compile date 2024-02-26 21:43:15 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: DB SUMMARY 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: DB Session ID: CL0UD0Y9W3WHAAJQE633 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: CURRENT file: CURRENT 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-02-27T07:52:44.216 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1625275 ; 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.error_if_exists: 0 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.create_if_missing: 0 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.paranoid_checks: 1 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.env: 0x55698ee81b20 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.fs: PosixFileSystem 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.info_log: 0x55698f8e58e0 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.statistics: (nil) 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.use_fsync: 0 2024-02-27T07:52:44.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_log_file_size: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_fallocate: 1 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.use_direct_reads: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.db_log_dir: 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.wal_dir: 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T07:52:44.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.write_buffer_manager: 0x55698f8b8dc0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T07:52:44.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.unordered_write: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.row_cache: None 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.wal_filter: None 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.two_write_queues: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.wal_compression: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.atomic_flush: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T07:52:44.220 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.log_readahead_size: 0 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T07:52:44.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_background_jobs: 2 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_background_compactions: -1 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_subcompactions: 1 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T07:52:44.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_open_files: -1 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_background_flushes: -1 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Compression algorithms supported: 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kZSTD supported: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kXpressCompression supported: 0 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kZlibCompression supported: 1 2024-02-27T07:52:44.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kSnappyCompression supported: 1 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kLZ4Compression supported: 1 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: kBZip2Compression supported: 0 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.merge_operator: 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_filter: None 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T07:52:44.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55698f8e59e0) 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: cache_index_and_filter_blocks: 1 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: pin_top_level_index_and_filter: 1 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_type: 0 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: data_block_index_type: 0 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_shortening: 1 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: checksum: 4 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: no_block_cache: 0 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache: 0x55698f8e9090 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_name: BinnedLRUCache 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_options: 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: capacity : 536870912 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: num_shard_bits : 4 2024-02-27T07:52:44.225 INFO:journalctl@ceph.mon.b.smithi195.stdout: strict_capacity_limit : 0 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: high_pri_pool_ratio: 0.000 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_compressed: (nil) 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: persistent_cache: (nil) 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_size: 4096 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_size_deviation: 10 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_restart_interval: 16 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_block_restart_interval: 1 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: metadata_block_size: 4096 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: partition_filters: 0 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: use_delta_encoding: 1 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: filter_policy: bloomfilter 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: whole_key_filtering: 1 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: verify_compression: 0 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: read_amp_bytes_per_bit: 0 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: format_version: 5 2024-02-27T07:52:44.226 INFO:journalctl@ceph.mon.b.smithi195.stdout: enable_index_compression: 1 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_align: 0 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout: max_auto_readahead_size: 262144 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout: prepopulate_block_cache: 0 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout: initial_auto_readahead_size: 8192 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout: num_file_reads_for_auto_readahead: 2 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression: NoCompression 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.num_levels: 7 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T07:52:44.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T07:52:44.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T07:52:44.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.arena_block_size: 1048576 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T07:52:44.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.table_properties_collectors: 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.inplace_update_support: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.bloom_locality: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.max_successive_merges: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.ttl: 2592000 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-27T07:52:44.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enable_blob_files: false 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.min_blob_size: 0 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.blob_file_starting_level: 0 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14709, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9eb0e594-730a-4ef5-8db9-0ded812a5cce 2024-02-27T07:52:44.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020363774797, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020363779452, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 971522, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14714, "largest_seqno": 15208, "table_properties": {"data_size": 968384, "index_size": 1548, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5156, "raw_average_key_size": 23, "raw_value_size": 963402, "raw_average_value_size": 4460, "num_data_blocks": 71, "num_entries": 216, "num_filter_entries": 216, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709020363, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9eb0e594-730a-4ef5-8db9-0ded812a5cce", "db_session_id": "CL0UD0Y9W3WHAAJQE633", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709020363779543, "job": 1, "event": "recovery_finished"} 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55698f9da000 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: DB pointer 0x55698f9c4000 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** DB Stats ** 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:44.233 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: L0 1/0 948.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 398.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: L6 1/0 11.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Sum 2/0 12.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 398.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 398.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:44.234 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** Compaction Stats [default] ** 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 398.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative compaction: 0.00 GB write, 134.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval compaction: 0.00 GB write, 134.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T07:52:44.235 INFO:journalctl@ceph.mon.b.smithi195.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: Block cache BinnedLRUCache@0x55698f8e9090#2 capacity: 512.00 MB usage: 2.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: starting mon.b rank 2 at public addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] at bind addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???) e3 preinit fsid 448a3a16-d543-11ee-95c3-87774f69a715 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).mds e1 new map 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).mds e1 print_map 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: e1 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: legacy client fscid: -1 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout: No filesystems configured 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-27T07:52:44.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:44.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:44.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T07:52:44.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 ceph-mon[58095]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-02-27T07:52:44.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 bash[58075]: 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 2024-02-27T07:52:44.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:43 smithi195 systemd[1]: Started Ceph mon.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:52:45.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:44] "GET /metrics HTTP/1.1" 200 38192 "" "Prometheus/2.33.4" 2024-02-27T07:52:48.965 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:48 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:50.226 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:52:50.877 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:52:50.877 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (2m) 17s ago 7m 17.7M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:52:50.877 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (101s) 32s ago 6m 43.5M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:52:50.877 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (71s) 17s ago 6m 48.9M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:52:50.877 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (61s) 32s ago 11m 446M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (96s) 17s ago 13m 490M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (47s) 17s ago 13m 40.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 starting - - - 2048M 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (21s) 17s ago 11m 20.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (2m) 17s ago 7m 17.3M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (117s) 32s ago 7m 19.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (11m) 17s ago 11m 53.2M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (10m) 17s ago 10m 53.4M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (10m) 17s ago 10m 48.3M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:52:50.878 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (10m) 17s ago 10m 52.4M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (9m) 32s ago 9m 53.7M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (9m) 32s ago 9m 49.5M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (8m) 32s ago 8m 48.8M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (8m) 32s ago 8m 51.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (67s) 32s ago 7m 55.4M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (6m) 17s ago 6m 71.8M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (6m) 32s ago 6m 71.7M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:52:50.879 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (6m) 17s ago 6m 73.7M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:52:50.880 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (6m) 32s ago 6m 73.5M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:52:51.269 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:52:51.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-27T07:52:51.271 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T07:52:51.271 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:52:51.271 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:52:51.602 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:52:51.602 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr" 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "4/23 daemons upgraded", 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading mon daemons", 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:52:51.603 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:52:51.603 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mon.b (rank 2) addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] is down (out of quorum) 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mon.b calling monitor election 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mon.c calling monitor election 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mon.a calling monitor election 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: monmap e4: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: fsmap 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:51.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: mgrmap e39: y(active, since 36s), standbys: x 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: overall HEALTH_OK 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2012386973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mon.b (rank 2) addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] is down (out of quorum) 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mon.b calling monitor election 2024-02-27T07:52:51.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mon.c calling monitor election 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mon.a calling monitor election 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: monmap e4: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: fsmap 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: mgrmap e39: y(active, since 36s), standbys: x 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: overall HEALTH_OK 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.606 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:51 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2012386973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='client.44154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mon.b (rank 2) addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] is down (out of quorum) 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mon.b calling monitor election 2024-02-27T07:52:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mon.c calling monitor election 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mon.a calling monitor election 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: monmap e4: 3 mons at {a=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: fsmap 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: mgrmap e39: y(active, since 36s), standbys: x 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: overall HEALTH_OK 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:51 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2012386973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:52:51.995 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:52:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:52 smithi195 ceph-mon[58095]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:52 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/628157205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:52 smithi195 ceph-mon[58095]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:52.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[67157]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:52.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/628157205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:52.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[67157]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[64590]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:52:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/628157205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:52:52.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:52 smithi149 ceph-mon[64590]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:54.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:53 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:52:53.787+0000 7fd6feeab700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-27T07:52:54.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:54 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:54.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:54 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:54.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:54 smithi195 ceph-mon[58095]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:55.090 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:52:54] "GET /metrics HTTP/1.1" 200 38192 "" "Prometheus/2.33.4" 2024-02-27T07:52:55.090 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.090 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.091 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[64590]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:55.091 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.091 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.091 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:54 smithi149 ceph-mon[67157]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:55.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:55.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:55 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:56.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:56.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:56.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:56.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:56.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:56.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:55 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:52:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:56 smithi195 ceph-mon[58095]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:57.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:56 smithi149 ceph-mon[67157]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:57.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:56 smithi149 ceph-mon[64590]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:52:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:52:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:52:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:52:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:59.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:59.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:52:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:52:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:52:59.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:52:59.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:52:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:00.423 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: Reconfiguring mon.a (monmap changed)... 2024-02-27T07:53:00.423 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:53:00.423 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.423 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.423 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: Reconfiguring mon.a (monmap changed)... 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:00.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: Reconfiguring mon.a (monmap changed)... 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: Reconfiguring daemon mon.a on smithi149 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[67157]: Reconfiguring mgr.y (monmap changed)... 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[67157]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[67157]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[64590]: Reconfiguring mgr.y (monmap changed)... 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[64590]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:53:01.267 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:01 smithi149 ceph-mon[64590]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:01 smithi195 ceph-mon[58095]: Reconfiguring mgr.y (monmap changed)... 2024-02-27T07:53:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:01 smithi195 ceph-mon[58095]: Reconfiguring daemon mgr.y on smithi149 2024-02-27T07:53:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:01 smithi195 ceph-mon[58095]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[64590]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.649 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:53:02.650 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:02 smithi149 ceph-mon[67157]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: Reconfiguring mon.c (monmap changed)... 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:02.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: Reconfiguring daemon mon.c on smithi149 2024-02-27T07:53:02.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:02 smithi195 ceph-mon[58095]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:04.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: Reconfiguring osd.0 (monmap changed)... 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.0 on smithi149 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: Reconfiguring osd.0 (monmap changed)... 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:04.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:03 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.0 on smithi149 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: Reconfiguring osd.0 (monmap changed)... 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:03 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.0 on smithi149 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:04] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.33.4" 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[67157]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:05.101 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[64590]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:05.102 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.102 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.102 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:53:05.102 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:04 smithi195 ceph-mon[58095]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:53:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:06.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[64590]: Reconfiguring osd.1 (monmap changed)... 2024-02-27T07:53:06.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.1 on smithi149 2024-02-27T07:53:06.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:06.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[67157]: Reconfiguring osd.1 (monmap changed)... 2024-02-27T07:53:06.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.1 on smithi149 2024-02-27T07:53:06.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:05 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:05 smithi195 ceph-mon[58095]: Reconfiguring osd.1 (monmap changed)... 2024-02-27T07:53:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:05 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.1 on smithi149 2024-02-27T07:53:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:05 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:07.156 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[64590]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:07.156 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.156 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[67157]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:53:07.157 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:07.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:06 smithi195 ceph-mon[58095]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:07.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:07.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:53:07.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:08.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: Reconfiguring osd.2 (monmap changed)... 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.2 on smithi149 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: Reconfiguring osd.2 (monmap changed)... 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.2 on smithi149 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:53:08.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:07 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: Reconfiguring osd.2 (monmap changed)... 2024-02-27T07:53:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.2 on smithi149 2024-02-27T07:53:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:53:08.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:07 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:08.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:09.057 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[64590]: Reconfiguring osd.3 (monmap changed)... 2024-02-27T07:53:09.057 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.3 on smithi149 2024-02-27T07:53:09.057 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[64590]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:09.057 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[67157]: Reconfiguring osd.3 (monmap changed)... 2024-02-27T07:53:09.058 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.3 on smithi149 2024-02-27T07:53:09.058 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:08 smithi149 ceph-mon[67157]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:09.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:08 smithi195 ceph-mon[58095]: Reconfiguring osd.3 (monmap changed)... 2024-02-27T07:53:09.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:08 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.3 on smithi149 2024-02-27T07:53:09.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:08 smithi195 ceph-mon[58095]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:10.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: Reconfiguring rgw.foo.smithi149.akhivq (monmap changed)... 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: Reconfiguring daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[67157]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: Reconfiguring rgw.foo.smithi149.akhivq (monmap changed)... 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: Reconfiguring daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:53:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:10 smithi149 ceph-mon[64590]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:10.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:10.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: Reconfiguring rgw.foo.smithi149.akhivq (monmap changed)... 2024-02-27T07:53:10.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:10.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:10.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: Reconfiguring daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:53:10.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:10 smithi195 ceph-mon[58095]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: Reconfiguring rgw.smpl.smithi149.bmexvh (monmap changed)... 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[67157]: Reconfiguring daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.328 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.329 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: Reconfiguring rgw.smpl.smithi149.bmexvh (monmap changed)... 2024-02-27T07:53:12.329 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:12.329 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:12.329 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:11 smithi149 ceph-mon[64590]: Reconfiguring daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: Reconfiguring rgw.smpl.smithi149.bmexvh (monmap changed)... 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:11 smithi195 ceph-mon[58095]: Reconfiguring daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:53:13.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:13.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:13.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:13.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:13.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T07:53:13.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T07:53:13.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:14.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:14 smithi195 ceph-mon[58095]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:53:14.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:14 smithi195 ceph-mon[58095]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:53:14.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:14 smithi149 ceph-mon[67157]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:53:14.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:14 smithi149 ceph-mon[67157]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:53:14.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:14 smithi149 ceph-mon[64590]: Reconfiguring mon.b (monmap changed)... 2024-02-27T07:53:14.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:14 smithi149 ceph-mon[64590]: Reconfiguring daemon mon.b on smithi195 2024-02-27T07:53:15.106 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:14] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.33.4" 2024-02-27T07:53:15.106 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:15.106 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:15.106 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.106 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:15.107 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:15.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:15.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:15.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:15.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T07:53:15.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T07:53:15.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:16 smithi195 ceph-mon[58095]: Reconfiguring mgr.x (monmap changed)... 2024-02-27T07:53:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:16 smithi195 ceph-mon[58095]: Reconfiguring daemon mgr.x on smithi195 2024-02-27T07:53:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:16 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:16 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:16.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:16 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:16.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[67157]: Reconfiguring mgr.x (monmap changed)... 2024-02-27T07:53:16.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[67157]: Reconfiguring daemon mgr.x on smithi195 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[64590]: Reconfiguring mgr.x (monmap changed)... 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[64590]: Reconfiguring daemon mgr.x on smithi195 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:16 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:17 smithi195 ceph-mon[58095]: Reconfiguring osd.4 (monmap changed)... 2024-02-27T07:53:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:53:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:17 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.4 on smithi195 2024-02-27T07:53:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:17 smithi195 ceph-mon[58095]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:17.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[67157]: Reconfiguring osd.4 (monmap changed)... 2024-02-27T07:53:17.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.4 on smithi195 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[67157]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[64590]: Reconfiguring osd.4 (monmap changed)... 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.4 on smithi195 2024-02-27T07:53:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:17 smithi149 ceph-mon[64590]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:18.875 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:18.875 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.875 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.875 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: Reconfiguring osd.5 (monmap changed)... 2024-02-27T07:53:18.876 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:53:18.876 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:18.876 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.5 on smithi195 2024-02-27T07:53:18.876 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:18 smithi195 ceph-mon[58095]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: Reconfiguring osd.5 (monmap changed)... 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.5 on smithi195 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[64590]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:18.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: Reconfiguring osd.5 (monmap changed)... 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.5 on smithi195 2024-02-27T07:53:18.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:18 smithi149 ceph-mon[67157]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: Reconfiguring osd.6 (monmap changed)... 2024-02-27T07:53:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:53:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:20.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.6 on smithi195 2024-02-27T07:53:20.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:20 smithi195 ceph-mon[58095]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:20.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: Reconfiguring osd.6 (monmap changed)... 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.6 on smithi195 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[64590]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: Reconfiguring osd.6 (monmap changed)... 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.6 on smithi195 2024-02-27T07:53:20.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:20 smithi149 ceph-mon[67157]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: Reconfiguring osd.7 (monmap changed)... 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:22.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:21 smithi195 ceph-mon[58095]: Reconfiguring daemon osd.7 on smithi195 2024-02-27T07:53:22.351 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: Reconfiguring osd.7 (monmap changed)... 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:22.351 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[67157]: Reconfiguring daemon osd.7 on smithi195 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: Reconfiguring osd.7 (monmap changed)... 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:22.352 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:21 smithi149 ceph-mon[64590]: Reconfiguring daemon osd.7 on smithi195 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (2m) 49s ago 7m 17.7M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (2m) 29s ago 7m 44.6M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (103s) 49s ago 6m 48.9M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (93s) 29s ago 12m 448M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (2m) 49s ago 13m 490M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (79s) 49s ago 13m 40.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (39s) 29s ago 12m 17.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (53s) 49s ago 12m 20.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (2m) 49s ago 8m 17.3M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:53:23.002 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (2m) 29s ago 8m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (11m) 49s ago 11m 53.2M 4096M 17.2.0 e1d6a67b021e eaaba4baf050 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (11m) 49s ago 11m 53.4M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (11m) 49s ago 11m 48.3M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (10m) 49s ago 10m 52.4M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (10m) 29s ago 10m 53.8M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (9m) 29s ago 9m 49.9M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (9m) 29s ago 9m 49.1M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (9m) 29s ago 9m 52.0M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (99s) 29s ago 8m 55.6M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (6m) 49s ago 6m 71.8M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (6m) 29s ago 6m 72.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (6m) 49s ago 6m 73.7M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:53:23.003 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (6m) 29s ago 6m 73.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:53:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:22 smithi195 ceph-mon[58095]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:23.390 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:53:23.390 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:53:23.390 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:53:23.390 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:53:23.391 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[67157]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[64590]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:23.392 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:23.393 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:23.740 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "5/23 daemons upgraded", 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading mon daemons", 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:53:23.741 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:53:24.152 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:53:24.152 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: Reconfiguring rgw.foo.smithi195.xuyrge (monmap changed)... 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: Reconfiguring daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/4265732614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: Reconfiguring rgw.foo.smithi195.xuyrge (monmap changed)... 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: Reconfiguring daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.153 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:23 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/4265732614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: Reconfiguring rgw.foo.smithi195.xuyrge (monmap changed)... 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: Reconfiguring daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:23 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/4265732614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:24.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:24] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.33.4" 2024-02-27T07:53:25.166 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:25.166 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:25.167 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3700441119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:25.167 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.167 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.167 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:25.167 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:25.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3700441119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3700441119' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:53:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: Reconfiguring rgw.smpl.smithi195.lvkuxs (monmap changed)... 2024-02-27T07:53:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: Reconfiguring daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:53:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:26.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: Reconfiguring rgw.smpl.smithi195.lvkuxs (monmap changed)... 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: Reconfiguring daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-27T07:53:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: Reconfiguring rgw.smpl.smithi195.lvkuxs (monmap changed)... 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: Reconfiguring daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:26.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:27.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all mon 2024-02-27T07:53:27.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all crash 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[67157]: Upgrade: osd.0 is safe to restart 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[67157]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all mon 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all crash 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[64590]: Upgrade: osd.0 is safe to restart 2024-02-27T07:53:27.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:26 smithi149 ceph-mon[64590]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:26 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all mon 2024-02-27T07:53:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:26 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all crash 2024-02-27T07:53:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:26 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T07:53:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:26 smithi195 ceph-mon[58095]: Upgrade: osd.0 is safe to restart 2024-02-27T07:53:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:26 smithi195 ceph-mon[58095]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: Upgrade: Updating osd.0 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: Deploying daemon osd.0 on smithi149 2024-02-27T07:53:28.599 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[67157]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: Upgrade: Updating osd.0 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: Deploying daemon osd.0 on smithi149 2024-02-27T07:53:28.600 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:28 smithi149 ceph-mon[64590]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:28.714 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: Upgrade: Updating osd.0 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: Deploying daemon osd.0 on smithi149 2024-02-27T07:53:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:28 smithi195 ceph-mon[58095]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:29.645 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:29.646 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:29.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:30.555 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:30 smithi149 ceph-mon[67157]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:30.555 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:30 smithi149 ceph-mon[64590]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:30.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:30 smithi195 ceph-mon[58095]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:30.844 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:30 smithi149 systemd[1]: Stopping Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:53:31.176 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[32383]: 2024-02-27T07:53:30.843+0000 7f148d850700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:53:31.176 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[32383]: 2024-02-27T07:53:30.843+0000 7f148d850700 -1 osd.0 105 *** Got signal Terminated *** 2024-02-27T07:53:31.177 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[32383]: 2024-02-27T07:53:30.843+0000 7f148d850700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:53:31.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:31 smithi149 ceph-mon[64590]: osd.0 marked itself down and dead 2024-02-27T07:53:31.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:31 smithi149 ceph-mon[67157]: osd.0 marked itself down and dead 2024-02-27T07:53:31.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:31 smithi195 ceph-mon[58095]: osd.0 marked itself down and dead 2024-02-27T07:53:32.277 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:31 smithi149 podman[70056]: 2024-02-27 07:53:31.937165735 +0000 UTC m=+1.217805119 container died eaaba4baf050bc8d94056165fa66437827e0b8d5238b5af4aafc6091db86358d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.expose-services=, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container) 2024-02-27T07:53:32.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:53:32.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[64590]: osdmap e106: 8 total, 7 up, 8 in 2024-02-27T07:53:32.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[64590]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:53:32.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:53:32.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[67157]: osdmap e106: 8 total, 7 up, 8 in 2024-02-27T07:53:32.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:32 smithi149 ceph-mon[67157]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:53:32.677 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:32 smithi149 podman[70056]: 2024-02-27 07:53:32.276617415 +0000 UTC m=+1.557256796 container cleanup eaaba4baf050bc8d94056165fa66437827e0b8d5238b5af4aafc6091db86358d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, io.openshift.expose-services=, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-type=git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-27T07:53:32.677 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:32 smithi149 bash[70056]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0 2024-02-27T07:53:32.678 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:32 smithi149 podman[70073]: 2024-02-27 07:53:32.444913243 +0000 UTC m=+0.509141525 container remove eaaba4baf050bc8d94056165fa66437827e0b8d5238b5af4aafc6091db86358d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., release=754, RELEASE=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-27T07:53:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:32 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:53:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:32 smithi195 ceph-mon[58095]: osdmap e106: 8 total, 7 up, 8 in 2024-02-27T07:53:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:32 smithi195 ceph-mon[58095]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:53:33.132 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:32 smithi149 podman[70164]: 2024-02-27 07:53:32.748406517 +0000 UTC m=+0.018744034 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:53:33.132 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:32 smithi149 podman[70164]: 2024-02-27 07:53:32.853825279 +0000 UTC m=+0.124162798 container create 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:53:33.132 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:33 smithi149 podman[70164]: 2024-02-27 07:53:33.05660791 +0000 UTC m=+0.326945440 container init 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-02-27T07:53:33.132 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:33 smithi149 podman[70164]: 2024-02-27 07:53:33.060214668 +0000 UTC m=+0.330552196 container start 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:53:33.132 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:33 smithi149 podman[70164]: 2024-02-27 07:53:33.131246497 +0000 UTC m=+0.401584016 container attach 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:53:33.426 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:33 smithi149 podman[70164]: 2024-02-27 07:53:33.350204803 +0000 UTC m=+0.620542313 container died 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-27T07:53:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:33 smithi195 ceph-mon[58095]: osdmap e107: 8 total, 7 up, 8 in 2024-02-27T07:53:33.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:33 smithi149 ceph-mon[67157]: osdmap e107: 8 total, 7 up, 8 in 2024-02-27T07:53:33.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:33 smithi149 ceph-mon[64590]: osdmap e107: 8 total, 7 up, 8 in 2024-02-27T07:53:34.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70189]: 2024-02-27 07:53:34.026699706 +0000 UTC m=+0.667660337 container remove 4970c5614d7d0b43cb18c72c89f7ef29c05e42521c27f6745b59dabfa48d091b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:53:34.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-27T07:53:34.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 systemd[1]: Stopped Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:53:34.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0.service: Consumed 6.837s CPU time. 2024-02-27T07:53:34.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 systemd[1]: Starting Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:53:34.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:34 smithi195 ceph-mon[58095]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:34.764 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:34 smithi149 ceph-mon[64590]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:34.764 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:34 smithi149 ceph-mon[67157]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:34.764 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70291]: 2024-02-27 07:53:34.374720514 +0000 UTC m=+0.018570807 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:53:34.764 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70291]: 2024-02-27 07:53:34.477848798 +0000 UTC m=+0.121699121 container create 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-27T07:53:34.765 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70291]: 2024-02-27 07:53:34.68899741 +0000 UTC m=+0.332847708 container init 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-02-27T07:53:34.765 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70291]: 2024-02-27 07:53:34.692372221 +0000 UTC m=+0.336222504 container start 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-02-27T07:53:34.765 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:34 smithi149 podman[70291]: 2024-02-27 07:53:34.763373328 +0000 UTC m=+0.407223613 container attach 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-27T07:53:35.126 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:34] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.33.4" 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-27T07:53:35.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 bash[70291]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate[70305]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-27T07:53:35.428 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 podman[70291]: 2024-02-27 07:53:35.161899428 +0000 UTC m=+0.805749699 container died 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-02-27T07:53:35.700 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:35.700 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:35 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:35.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:35 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:36.033 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 podman[70442]: 2024-02-27 07:53:35.70024358 +0000 UTC m=+0.533117787 container remove 5d6842acc82cc619e1b28b73bfff10379ceaa3eca59fb30efba1298dd07b7623 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T07:53:36.033 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 podman[70508]: 2024-02-27 07:53:35.816869919 +0000 UTC m=+0.017631295 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:53:36.033 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:35 smithi149 podman[70508]: 2024-02-27 07:53:35.934641023 +0000 UTC m=+0.135402387 container create 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5) 2024-02-27T07:53:36.426 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 podman[70508]: 2024-02-27 07:53:36.145686286 +0000 UTC m=+0.346447670 container init 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-27T07:53:36.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 podman[70508]: 2024-02-27 07:53:36.148926362 +0000 UTC m=+0.349687727 container start 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-27T07:53:36.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-osd[70527]: -- 172.21.15.149:0/345216818 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594a8ead680 con 0x5594a8e77000 2024-02-27T07:53:36.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 bash[70508]: 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac 2024-02-27T07:53:36.427 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 systemd[1]: Started Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:53:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:36 smithi195 ceph-mon[58095]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:36.926 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T07:53:36.698+0000 7fe3efbba7c0 -1 Falling back to public interface 2024-02-27T07:53:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[64590]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:36.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:36.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[67157]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:36.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:36.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:37.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:37 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2024-02-27T07:53:37.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:37 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2024-02-27T07:53:37.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:37 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2024-02-27T07:53:38.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:38 smithi195 ceph-mon[58095]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:38.714 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:38.788 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:38 smithi149 ceph-mon[67157]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:38.788 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:38 smithi149 ceph-mon[64590]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:39.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-mon[67157]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:39.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-mon[67157]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:39.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-mon[64590]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:39.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-mon[64590]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:39.677 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T07:53:39.381+0000 7fe3efbba7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-27T07:53:39.677 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T07:53:39.406+0000 7fe3efbba7c0 -1 osd.0 105 log_to_monitors true 2024-02-27T07:53:39.677 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 07:53:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T07:53:39.504+0000 7fe3e1012700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:53:39.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:39 smithi195 ceph-mon[58095]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:39.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:39 smithi195 ceph-mon[58095]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T07:53:40.593 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[67157]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[67157]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[67157]: osdmap e108: 8 total, 7 up, 8 in 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[67157]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[67157]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[64590]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[64590]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[64590]: osdmap e108: 8 total, 7 up, 8 in 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[64590]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.594 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:40 smithi149 ceph-mon[64590]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:40 smithi195 ceph-mon[58095]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T07:53:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:40 smithi195 ceph-mon[58095]: from='osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:40 smithi195 ceph-mon[58095]: osdmap e108: 8 total, 7 up, 8 in 2024-02-27T07:53:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:40 smithi195 ceph-mon[58095]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:53:40.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:40 smithi195 ceph-mon[58095]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2024-02-27T07:53:41.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:53:41.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212] boot 2024-02-27T07:53:41.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: osdmap e109: 8 total, 8 up, 8 in 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212] boot 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: osdmap e109: 8 total, 8 up, 8 in 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: osd.0 [v2:172.21.15.149:6802/3035596212,v1:172.21.15.149:6803/3035596212] boot 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: osdmap e109: 8 total, 8 up, 8 in 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.774 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[64590]: osdmap e110: 8 total, 8 up, 8 in 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[64590]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[67157]: osdmap e110: 8 total, 8 up, 8 in 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.775 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:42 smithi149 ceph-mon[67157]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:42 smithi195 ceph-mon[58095]: osdmap e110: 8 total, 8 up, 8 in 2024-02-27T07:53:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:42 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:42 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:42 smithi195 ceph-mon[58095]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:44.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:44] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.33.4" 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[64590]: pgmap v53: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[67157]: pgmap v53: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:45 smithi195 ceph-mon[58095]: pgmap v53: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2024-02-27T07:53:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:45.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:45 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:53:45.700+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:53:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:46.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:47.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[67157]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[64590]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2024-02-27T07:53:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:47 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:53:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:47 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:53:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:47 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-02-27T07:53:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:47 smithi195 ceph-mon[58095]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:53:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:47 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2024-02-27T07:53:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:47 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:53:48.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:48 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:49.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:49 smithi149 ceph-mon[67157]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:49.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:49 smithi149 ceph-mon[64590]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:49 smithi195 ceph-mon[58095]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:53:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:51 smithi149 ceph-mon[67157]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:53:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:51 smithi149 ceph-mon[64590]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:53:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:51 smithi195 ceph-mon[58095]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:53:53.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:53 smithi149 ceph-mon[64590]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-02-27T07:53:53.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:53 smithi149 ceph-mon[67157]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-02-27T07:53:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:53 smithi195 ceph-mon[58095]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-02-27T07:53:54.518 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:53:55.128 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:53:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:53:54] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.33.4" 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (3m) 14s ago 8m 26.2M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (2m) 61s ago 7m 44.6M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (2m) 14s ago 7m 52.0M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (2m) 61s ago 12m 448M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (2m) 14s ago 14m 498M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (111s) 14s ago 14m 47.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (71s) 61s ago 12m 17.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (86s) 14s ago 13m 37.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:53:55.218 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (3m) 14s ago 8m 16.9M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (3m) 61s ago 8m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (19s) 14s ago 12m 12.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (11m) 14s ago 11m 54.5M 4096M 17.2.0 e1d6a67b021e 09c5c1ddecad 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (11m) 14s ago 11m 49.2M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (11m) 14s ago 11m 53.6M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (10m) 61s ago 10m 53.8M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (10m) 61s ago 10m 49.9M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (10m) 61s ago 10m 49.1M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (9m) 61s ago 9m 52.0M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (2m) 61s ago 8m 55.6M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (7m) 14s ago 7m 72.1M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (7m) 61s ago 7m 72.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (7m) 14s ago 7m 74.2M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:53:55.219 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (7m) 61s ago 7m 73.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:53:55.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:55 smithi149 ceph-mon[67157]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:55.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:55 smithi149 ceph-mon[64590]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:55 smithi195 ceph-mon[58095]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:55.617 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 6 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:53:55.618 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:53:55.946 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:53:55.946 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:53:55.946 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:53:55.946 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:53:55.946 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "6/23 daemons upgraded", 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:53:55.947 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:53:56.346 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:53:56.346 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[67157]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.346 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[67157]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.346 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:56.347 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3337741477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:56.347 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[64590]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.347 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[64590]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.347 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:56.347 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:56 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3337741477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:56 smithi195 ceph-mon[58095]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:56 smithi195 ceph-mon[58095]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:56 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:53:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:56 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3337741477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:53:57.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[64590]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[64590]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[64590]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3477228852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[67157]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[67157]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[67157]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:57 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3477228852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:57 smithi195 ceph-mon[58095]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:57 smithi195 ceph-mon[58095]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:53:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:57 smithi195 ceph-mon[58095]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:53:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:57 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3477228852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:53:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:53:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:53:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:53:59.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[64590]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:59.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:59.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[67157]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:59.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:53:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:53:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:59 smithi195 ceph-mon[58095]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:53:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:53:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:53:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:01.393 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:01 smithi149 ceph-mon[67157]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:01.394 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:01 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:01.394 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:01 smithi149 ceph-mon[64590]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:01.394 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:01 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:01 smithi195 ceph-mon[58095]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:01 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:02.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:02 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:02.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:02 smithi149 ceph-mon[64590]: Upgrade: osd.1 is safe to restart 2024-02-27T07:54:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:02 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:02 smithi149 ceph-mon[67157]: Upgrade: osd.1 is safe to restart 2024-02-27T07:54:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:02 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T07:54:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:02 smithi195 ceph-mon[58095]: Upgrade: osd.1 is safe to restart 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[67157]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[64590]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:54:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:03 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:03 smithi195 ceph-mon[58095]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T07:54:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:03 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:04.403 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:04 smithi149 ceph-mon[67157]: Upgrade: Updating osd.1 2024-02-27T07:54:04.403 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:04 smithi149 ceph-mon[67157]: Deploying daemon osd.1 on smithi149 2024-02-27T07:54:04.403 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:04 smithi149 ceph-mon[64590]: Upgrade: Updating osd.1 2024-02-27T07:54:04.403 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:04 smithi149 ceph-mon[64590]: Deploying daemon osd.1 on smithi149 2024-02-27T07:54:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:04 smithi195 ceph-mon[58095]: Upgrade: Updating osd.1 2024-02-27T07:54:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:04 smithi195 ceph-mon[58095]: Deploying daemon osd.1 on smithi149 2024-02-27T07:54:04.998 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:04] "GET /metrics HTTP/1.1" 200 38354 "" "Prometheus/2.33.4" 2024-02-27T07:54:05.365 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:05 smithi149 ceph-mon[67157]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:05.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:05 smithi149 ceph-mon[64590]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:05 smithi195 ceph-mon[58095]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:05.661 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:05 smithi149 systemd[1]: Stopping Ceph osd.1 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:54:05.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[35367]: 2024-02-27T07:54:05.659+0000 7fc51abd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:54:05.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[35367]: 2024-02-27T07:54:05.659+0000 7fc51abd3700 -1 osd.1 110 *** Got signal Terminated *** 2024-02-27T07:54:05.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:05 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[35367]: 2024-02-27T07:54:05.659+0000 7fc51abd3700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[67157]: osd.1 marked itself down and dead 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[67157]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[64590]: osd.1 marked itself down and dead 2024-02-27T07:54:06.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:06 smithi149 ceph-mon[64590]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:06 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:06 smithi195 ceph-mon[58095]: osd.1 marked itself down and dead 2024-02-27T07:54:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:06 smithi195 ceph-mon[58095]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:06.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:06 smithi149 podman[72530]: 2024-02-27 07:54:06.658507692 +0000 UTC m=+1.122510964 container died 09c5c1ddecadb9dc5dcb866b930bf46c284f06059ba8745cac982058df2b2118 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-02-27T07:54:07.319 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:07 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:07.320 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:07 smithi149 ceph-mon[67157]: osdmap e111: 8 total, 7 up, 8 in 2024-02-27T07:54:07.320 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:07 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:07.320 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:07 smithi149 ceph-mon[64590]: osdmap e111: 8 total, 7 up, 8 in 2024-02-27T07:54:07.320 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72530]: 2024-02-27 07:54:07.018039884 +0000 UTC m=+1.482043161 container cleanup 09c5c1ddecadb9dc5dcb866b930bf46c284f06059ba8745cac982058df2b2118 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, distribution-scope=public, io.openshift.tags=base centos centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.expose-services=, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-27T07:54:07.320 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 bash[72530]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1 2024-02-27T07:54:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:07 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:07 smithi195 ceph-mon[58095]: osdmap e111: 8 total, 7 up, 8 in 2024-02-27T07:54:07.676 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72550]: 2024-02-27 07:54:07.319379323 +0000 UTC m=+0.660915539 container remove 09c5c1ddecadb9dc5dcb866b930bf46c284f06059ba8745cac982058df2b2118 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, version=8, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, RELEASE=HEAD) 2024-02-27T07:54:07.981 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72641]: 2024-02-27 07:54:07.590410661 +0000 UTC m=+0.018639817 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:07.981 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72641]: 2024-02-27 07:54:07.723559484 +0000 UTC m=+0.151788679 container create b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-02-27T07:54:07.981 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72641]: 2024-02-27 07:54:07.914355837 +0000 UTC m=+0.342585000 container init b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-27T07:54:07.981 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72641]: 2024-02-27 07:54:07.917809055 +0000 UTC m=+0.346038210 container start b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-27T07:54:07.981 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:07 smithi149 podman[72641]: 2024-02-27 07:54:07.980706985 +0000 UTC m=+0.408936135 container attach b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-02-27T07:54:08.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:08 smithi149 ceph-mon[67157]: osdmap e112: 8 total, 7 up, 8 in 2024-02-27T07:54:08.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:08 smithi149 ceph-mon[67157]: pgmap v67: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:08.427 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:08 smithi149 podman[72641]: 2024-02-27 07:54:08.212006866 +0000 UTC m=+0.640236015 container died b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True) 2024-02-27T07:54:08.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:08 smithi149 ceph-mon[64590]: osdmap e112: 8 total, 7 up, 8 in 2024-02-27T07:54:08.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:08 smithi149 ceph-mon[64590]: pgmap v67: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:08 smithi195 ceph-mon[58095]: osdmap e112: 8 total, 7 up, 8 in 2024-02-27T07:54:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:08 smithi195 ceph-mon[58095]: pgmap v67: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:08.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:09.176 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:08 smithi149 podman[72668]: 2024-02-27 07:54:08.800170339 +0000 UTC m=+0.582505596 container remove b5c22fb42ccf359724690d37c723f1b4c00918afa643d066fc9f55c58913265d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:54:09.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:08 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-27T07:54:09.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:08 smithi149 systemd[1]: Stopped Ceph osd.1 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:54:09.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:08 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.1.service: Consumed 7.469s CPU time. 2024-02-27T07:54:09.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 systemd[1]: Starting Ceph osd.1 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:54:09.512 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.155541776 +0000 UTC m=+0.019883288 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:09.512 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.276393519 +0000 UTC m=+0.140735073 container create 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-02-27T07:54:09.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.512134426 +0000 UTC m=+0.376475939 container init 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-27T07:54:09.927 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.515457512 +0000 UTC m=+0.379799031 container start 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-02-27T07:54:09.927 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.603774329 +0000 UTC m=+0.468115861 container attach 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-02-27T07:54:10.290 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:10 smithi149 ceph-mon[67157]: pgmap v68: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:10.290 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:10 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:10.291 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:10 smithi149 ceph-mon[64590]: pgmap v68: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:10.291 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:10 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-27T07:54:10.291 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 bash[72773]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate[72788]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-27T07:54:10.292 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:09 smithi149 podman[72773]: 2024-02-27 07:54:09.985475118 +0000 UTC m=+0.849816625 container died 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:54:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:10 smithi195 ceph-mon[58095]: pgmap v68: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:10 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:10.889 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:10 smithi149 podman[72924]: 2024-02-27 07:54:10.574789164 +0000 UTC m=+0.583858139 container remove 02812423daad0004234a603a5c6ccf553487b04abbeba9f7e0b96ad83ba5bc7a (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-02-27T07:54:10.889 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:10 smithi149 podman[72989]: 2024-02-27 07:54:10.775515483 +0000 UTC m=+0.098556442 container create 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-02-27T07:54:10.889 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:10 smithi149 podman[72989]: 2024-02-27 07:54:10.694935668 +0000 UTC m=+0.017976580 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:11.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 podman[72989]: 2024-02-27 07:54:11.010956407 +0000 UTC m=+0.333997305 container init 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-27T07:54:11.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 podman[72989]: 2024-02-27 07:54:11.014306903 +0000 UTC m=+0.337347801 container start 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-27T07:54:11.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 ceph-osd[73008]: -- 172.21.15.149:0/1455259441 <== mon.1 v2:172.21.15.149:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56016333b680 con 0x560163305000 2024-02-27T07:54:11.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 bash[72989]: 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 2024-02-27T07:54:11.177 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 systemd[1]: Started Ceph osd.1 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:54:11.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:11 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T07:54:11.551+0000 7f0d132f07c0 -1 Falling back to public interface 2024-02-27T07:54:12.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:12 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:12.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:12 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:12 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:12.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:12 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:12 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:12 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:13.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:13 smithi149 ceph-mon[67157]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:13 smithi149 ceph-mon[64590]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:13 smithi195 ceph-mon[58095]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:14.676 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T07:54:14.258+0000 7f0d132f07c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-27T07:54:14.676 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T07:54:14.286+0000 7f0d132f07c0 -1 osd.1 110 log_to_monitors true 2024-02-27T07:54:15.132 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:14] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.33.4" 2024-02-27T07:54:15.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[67157]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:15.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:15.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:15.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[67157]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:54:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[64590]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-mon[64590]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:54:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:15 smithi195 ceph-mon[58095]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:15 smithi195 ceph-mon[58095]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T07:54:15.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 07:54:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T07:54:15.612+0000 7f0d00740700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:54:16.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: osdmap e113: 8 total, 7 up, 8 in 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[67157]: pgmap v72: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:54:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: osdmap e113: 8 total, 7 up, 8 in 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' 2024-02-27T07:54:16.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:16 smithi149 ceph-mon[64590]: pgmap v72: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: osdmap e113: 8 total, 7 up, 8 in 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: from='osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339]' entity='osd.1' 2024-02-27T07:54:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:16 smithi195 ceph-mon[58095]: pgmap v72: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339] boot 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: osdmap e114: 8 total, 8 up, 8 in 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339] boot 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: osdmap e114: 8 total, 8 up, 8 in 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:17.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: osd.1 [v2:172.21.15.149:6810/607648339,v1:172.21.15.149:6811/607648339] boot 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: osdmap e114: 8 total, 8 up, 8 in 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:18.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:18 smithi149 ceph-mon[64590]: osdmap e115: 8 total, 8 up, 8 in 2024-02-27T07:54:18.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:18 smithi149 ceph-mon[64590]: pgmap v75: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/630 objects degraded (14.127%) 2024-02-27T07:54:18.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:18 smithi149 ceph-mon[67157]: osdmap e115: 8 total, 8 up, 8 in 2024-02-27T07:54:18.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:18 smithi149 ceph-mon[67157]: pgmap v75: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/630 objects degraded (14.127%) 2024-02-27T07:54:18.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:18 smithi195 ceph-mon[58095]: osdmap e115: 8 total, 8 up, 8 in 2024-02-27T07:54:18.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:18 smithi195 ceph-mon[58095]: pgmap v75: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/630 objects degraded (14.127%) 2024-02-27T07:54:18.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:18.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-27T07:54:18.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:20.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:54:20.269+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 12 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/630 objects degraded (5.079%) 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 12 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/630 objects degraded (5.079%) 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 12 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/630 objects degraded (5.079%) 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:21.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:21.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:22.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:22 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:22 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T07:54:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:22 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:22 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T07:54:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:22 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:22 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T07:54:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[64590]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-02-27T07:54:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded) 2024-02-27T07:54:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:54:23.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[67157]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-02-27T07:54:23.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded) 2024-02-27T07:54:23.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:23 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:54:23.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:23 smithi195 ceph-mon[58095]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-02-27T07:54:23.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:23 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 7 pgs degraded) 2024-02-27T07:54:23.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:23 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:54:25.121 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:24] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.33.4" 2024-02-27T07:54:25.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:25 smithi149 ceph-mon[64590]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:25 smithi149 ceph-mon[67157]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:25.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:25 smithi195 ceph-mon[58095]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:26.703 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:54:27.344 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:54:27.344 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (3m) 11s ago 8m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:54:27.344 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (3m) 93s ago 8m 44.6M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:54:27.344 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (2m) 11s ago 7m 52.2M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:54:27.344 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (2m) 93s ago 13m 448M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (3m) 11s ago 14m 503M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (2m) 11s ago 14m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (103s) 93s ago 13m 17.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (118s) 11s ago 13m 39.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (4m) 11s ago 9m 16.8M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (3m) 93s ago 9m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (51s) 11s ago 12m 42.9M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (16s) 11s ago 12m 12.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (12m) 11s ago 12m 50.7M 4096M 17.2.0 e1d6a67b021e 92c4c79b5bf1 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (11m) 11s ago 11m 54.3M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (11m) 93s ago 11m 53.8M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (10m) 93s ago 10m 49.9M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (10m) 93s ago 10m 49.1M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:54:27.345 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (10m) 93s ago 10m 52.0M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:54:27.346 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (2m) 93s ago 9m 55.6M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:54:27.346 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (8m) 11s ago 8m 72.3M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:54:27.346 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (7m) 93s ago 7m 72.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:54:27.346 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (7m) 11s ago 7m 74.3M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:54:27.346 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (7m) 93s ago 7m 73.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:54:27.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:27 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:27.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:27 smithi149 ceph-mon[64590]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:54:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:27 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:27.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:27 smithi149 ceph-mon[67157]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:54:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:27 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:27 smithi195 ceph-mon[58095]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:54:27.732 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 7 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:54:27.733 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:54:28.061 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "7/23 daemons upgraded", 2024-02-27T07:54:28.062 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:54:28.062 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:54:28.062 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:54:28.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[67157]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[67157]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3265515822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:28.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[64590]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[64590]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:28 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3265515822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:28.452 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:54:28.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:28 smithi195 ceph-mon[58095]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:28 smithi195 ceph-mon[58095]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:28.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:28 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3265515822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:28.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-27T07:54:28.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:29.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[67157]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[67157]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[67157]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3214064599' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[64590]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[64590]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[64590]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-02-27T07:54:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:29 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3214064599' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:54:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:29 smithi195 ceph-mon[58095]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:29 smithi195 ceph-mon[58095]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:54:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:29 smithi195 ceph-mon[58095]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-02-27T07:54:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:29 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3214064599' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:31 smithi149 ceph-mon[67157]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:31 smithi149 ceph-mon[64590]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:31 smithi195 ceph-mon[58095]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:32 smithi195 ceph-mon[58095]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:32.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:32 smithi149 ceph-mon[67157]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:32.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:32 smithi149 ceph-mon[64590]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:34.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:34] "GET /metrics HTTP/1.1" 200 38443 "" "Prometheus/2.33.4" 2024-02-27T07:54:35.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:35 smithi149 ceph-mon[64590]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:35 smithi149 ceph-mon[67157]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:35 smithi195 ceph-mon[58095]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:36.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:36.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:37.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: Upgrade: osd.2 is safe to restart 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: Upgrade: osd.2 is safe to restart 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:54:37.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:37 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: Upgrade: osd.2 is safe to restart 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T07:54:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:37 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:38.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:38 smithi149 ceph-mon[64590]: Upgrade: Updating osd.2 2024-02-27T07:54:38.301 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:38 smithi149 ceph-mon[64590]: Deploying daemon osd.2 on smithi149 2024-02-27T07:54:38.302 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:38 smithi149 ceph-mon[67157]: Upgrade: Updating osd.2 2024-02-27T07:54:38.302 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:38 smithi149 ceph-mon[67157]: Deploying daemon osd.2 on smithi149 2024-02-27T07:54:38.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:38 smithi195 ceph-mon[58095]: Upgrade: Updating osd.2 2024-02-27T07:54:38.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:38 smithi195 ceph-mon[58095]: Deploying daemon osd.2 on smithi149 2024-02-27T07:54:38.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:39.261 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:39 smithi149 ceph-mon[64590]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:39.261 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:39 smithi149 ceph-mon[67157]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:39 smithi195 ceph-mon[58095]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:54:40.238 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:40 smithi149 ceph-mon[64590]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:40.238 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:40 smithi149 ceph-mon[67157]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:40.238 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:39 smithi149 systemd[1]: Stopping Ceph osd.2 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:54:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:40 smithi195 ceph-mon[58095]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:54:40.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:40 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[38288]: 2024-02-27T07:54:40.271+0000 7f58165a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:54:40.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:40 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[38288]: 2024-02-27T07:54:40.271+0000 7f58165a0700 -1 osd.2 115 *** Got signal Terminated *** 2024-02-27T07:54:40.677 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:40 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[38288]: 2024-02-27T07:54:40.271+0000 7f58165a0700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:54:41.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:41 smithi149 ceph-mon[64590]: osd.2 marked itself down and dead 2024-02-27T07:54:41.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:41 smithi149 ceph-mon[67157]: osd.2 marked itself down and dead 2024-02-27T07:54:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:41 smithi195 ceph-mon[58095]: osd.2 marked itself down and dead 2024-02-27T07:54:41.857 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:41 smithi149 podman[75015]: 2024-02-27 07:54:41.569319809 +0000 UTC m=+1.447395952 container died 92c4c79b5bf15e5b63d1b9d5fcf9a622d6158c859f47868d24309fde0437e6be (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-type=git) 2024-02-27T07:54:42.167 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:41 smithi149 podman[75015]: 2024-02-27 07:54:41.954514856 +0000 UTC m=+1.832591015 container cleanup 92c4c79b5bf15e5b63d1b9d5fcf9a622d6158c859f47868d24309fde0437e6be (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, vcs-type=git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, version=8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-02-27T07:54:42.167 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:41 smithi149 bash[75015]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2 2024-02-27T07:54:42.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[64590]: osdmap e116: 8 total, 7 up, 8 in 2024-02-27T07:54:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[64590]: pgmap v88: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:54:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[67157]: osdmap e116: 8 total, 7 up, 8 in 2024-02-27T07:54:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:42 smithi149 ceph-mon[67157]: pgmap v88: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:54:42.427 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75033]: 2024-02-27 07:54:42.272625292 +0000 UTC m=+0.700649142 container remove 92c4c79b5bf15e5b63d1b9d5fcf9a622d6158c859f47868d24309fde0437e6be (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, vcs-type=git, version=8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, architecture=x86_64, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD) 2024-02-27T07:54:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:42 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:54:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:42 smithi195 ceph-mon[58095]: osdmap e116: 8 total, 7 up, 8 in 2024-02-27T07:54:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:42 smithi195 ceph-mon[58095]: pgmap v88: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:54:42.859 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75127]: 2024-02-27 07:54:42.478727313 +0000 UTC m=+0.017819571 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:42.859 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75127]: 2024-02-27 07:54:42.590154761 +0000 UTC m=+0.129247055 container create 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.schema-version=1.0) 2024-02-27T07:54:42.860 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75127]: 2024-02-27 07:54:42.784536782 +0000 UTC m=+0.323629034 container init 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-02-27T07:54:42.860 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75127]: 2024-02-27 07:54:42.787703709 +0000 UTC m=+0.326795968 container start 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-27T07:54:42.860 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:42 smithi149 podman[75127]: 2024-02-27 07:54:42.859315589 +0000 UTC m=+0.398407850 container attach 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-27T07:54:43.176 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 podman[75127]: 2024-02-27 07:54:43.091129284 +0000 UTC m=+0.630221547 container died 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:54:43.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:43 smithi195 ceph-mon[58095]: osdmap e117: 8 total, 7 up, 8 in 2024-02-27T07:54:43.604 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:43 smithi149 ceph-mon[64590]: osdmap e117: 8 total, 7 up, 8 in 2024-02-27T07:54:43.605 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:43 smithi149 ceph-mon[67157]: osdmap e117: 8 total, 7 up, 8 in 2024-02-27T07:54:43.926 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 podman[75153]: 2024-02-27 07:54:43.6045383 +0000 UTC m=+0.507916587 container remove 64e8f5f90a225418a8131a2a9c6477d1217a0b080af1be0afb59f0a88db9bd8e (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-27T07:54:43.927 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-27T07:54:43.927 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 systemd[1]: Stopped Ceph osd.2 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:54:43.927 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.2.service: Consumed 6.522s CPU time. 2024-02-27T07:54:43.927 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:43 smithi149 systemd[1]: Starting Ceph osd.2 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:54:44.325 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:44.063707995 +0000 UTC m=+0.114632188 container create dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T07:54:44.325 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:43.968565574 +0000 UTC m=+0.019489787 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:44.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:44.324789795 +0000 UTC m=+0.375713988 container init dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-27T07:54:44.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:44.328172131 +0000 UTC m=+0.379096326 container start dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:54:44.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:44.408053452 +0000 UTC m=+0.458977646 container attach dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-27T07:54:45.113 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:44] "GET /metrics HTTP/1.1" 200 38441 "" "Prometheus/2.33.4" 2024-02-27T07:54:45.113 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 bash[75260]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate[75275]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-27T07:54:45.114 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:44 smithi149 podman[75260]: 2024-02-27 07:54:44.797838415 +0000 UTC m=+0.848762608 container died dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-27T07:54:45.387 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[67157]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:45.387 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:45.387 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:45.387 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[64590]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:45.388 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:45.388 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:45 smithi195 ceph-mon[58095]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:54:45.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 podman[75397]: 2024-02-27 07:54:45.386960855 +0000 UTC m=+0.583729928 container remove dcc463d9cd6516cbd3f300eeec095c654cf9eafcac397bd27474cd693a2f7a09 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:54:45.677 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 podman[75463]: 2024-02-27 07:54:45.506052481 +0000 UTC m=+0.018410931 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:54:45.677 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 podman[75463]: 2024-02-27 07:54:45.612920314 +0000 UTC m=+0.125278759 container create 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:54:46.176 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 podman[75463]: 2024-02-27 07:54:45.83174823 +0000 UTC m=+0.344106673 container init 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-27T07:54:46.177 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 podman[75463]: 2024-02-27 07:54:45.834837613 +0000 UTC m=+0.347196056 container start 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, CEPH_POINT_RELEASE=) 2024-02-27T07:54:46.177 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 ceph-osd[75482]: -- 172.21.15.149:0/4274280130 <== mon.0 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561bf8ed5680 con 0x561bf8e9ec00 2024-02-27T07:54:46.177 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 bash[75463]: 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 2024-02-27T07:54:46.177 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:45 smithi149 systemd[1]: Started Ceph osd.2 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:54:46.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T07:54:46.383+0000 7faa43c657c0 -1 Falling back to public interface 2024-02-27T07:54:47.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:47.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[67157]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:46 smithi149 ceph-mon[64590]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:46 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:46 smithi195 ceph-mon[58095]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:48.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:47 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:48.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:47 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:48.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:47 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:54:48.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:48 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-27T07:54:48.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:48 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:49.319 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:48 smithi149 ceph-mon[64590]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:49.320 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:48 smithi149 ceph-mon[67157]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:49.320 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T07:54:49.096+0000 7faa43c657c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-27T07:54:49.320 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T07:54:49.123+0000 7faa43c657c0 -1 osd.2 115 log_to_monitors true 2024-02-27T07:54:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:48 smithi195 ceph-mon[58095]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:50.298 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 07:54:50 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T07:54:50.006+0000 7faa350bd700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:54:50.299 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-mon[64590]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:50.299 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-mon[64590]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:50.299 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-mon[67157]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:50.299 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:49 smithi149 ceph-mon[67157]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:49 smithi195 ceph-mon[58095]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:49 smithi195 ceph-mon[58095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T07:54:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:54:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: osdmap e118: 8 total, 7 up, 8 in 2024-02-27T07:54:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: osdmap e118: 8 total, 7 up, 8 in 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: osdmap e118: 8 total, 7 up, 8 in 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: from='osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.287 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:52.287 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511] boot 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: osdmap e119: 8 total, 8 up, 8 in 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[67157]: osdmap e120: 8 total, 8 up, 8 in 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511] boot 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: osdmap e119: 8 total, 8 up, 8 in 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.288 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:52 smithi149 ceph-mon[64590]: osdmap e120: 8 total, 8 up, 8 in 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: osd.2 [v2:172.21.15.149:6818/3106117511,v1:172.21.15.149:6819/3106117511] boot 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: osdmap e119: 8 total, 8 up, 8 in 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:52 smithi195 ceph-mon[58095]: osdmap e120: 8 total, 8 up, 8 in 2024-02-27T07:54:53.290 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:53 smithi149 ceph-mon[67157]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:53.290 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:53 smithi149 ceph-mon[64590]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:53 smithi195 ceph-mon[58095]: pgmap v97: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:54.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:54:54] "GET /metrics HTTP/1.1" 200 38441 "" "Prometheus/2.33.4" 2024-02-27T07:54:55.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:54:55.196+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:55.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:55 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 31 active+undersized, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-02-27T07:54:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:54:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:54:55.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:55 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:54:56.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:54:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:54:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:57.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded) 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded) 2024-02-27T07:54:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:57 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded) 2024-02-27T07:54:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:57 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:54:58.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:54:58 smithi149 ceph-mon[67157]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:54:58 smithi149 ceph-mon[64590]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:58.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:54:58 smithi195 ceph-mon[58095]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:54:58.805 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:54:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-02-27T07:54:58.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:54:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:54:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:54:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:54:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (4m) 8s ago 9m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:54:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (3m) 2m ago 8m 44.6M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:54:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (3m) 8s ago 8m 52.4M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:54:59.465 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (3m) 2m ago 13m 448M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (3m) 8s ago 15m 505M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (2m) 8s ago 15m 51.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (2m) 2m ago 13m 17.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (2m) 8s ago 14m 41.7M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (4m) 8s ago 9m 17.2M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (4m) 2m ago 9m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (83s) 8s ago 13m 43.4M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (48s) 8s ago 12m 52.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (13s) 8s ago 12m 11.9M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (12m) 8s ago 12m 55.8M 4096M 17.2.0 e1d6a67b021e ab481c2a7928 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (11m) 2m ago 11m 53.8M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (11m) 2m ago 11m 49.9M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (11m) 2m ago 11m 49.1M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (10m) 2m ago 10m 52.0M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (3m) 2m ago 10m 55.6M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:54:59.466 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (8m) 8s ago 8m 72.4M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:54:59.467 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (8m) 2m ago 8m 72.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:54:59.467 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (8m) 8s ago 8m 74.5M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:54:59.467 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (8m) 2m ago 8m 73.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:54:59.854 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:54:59.855 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:55:00.190 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:55:00.191 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:55:00.191 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "8/23 daemons upgraded", 2024-02-27T07:55:00.191 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:55:00.191 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:55:00.191 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[67157]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[67157]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2599896313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[64590]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[64590]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:00 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2599896313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:00 smithi195 ceph-mon[58095]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:00 smithi195 ceph-mon[58095]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:00 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2599896313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:00.581 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:55:01.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[64590]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:01.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[64590]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:55:01.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/4079606771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[67157]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[67157]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:55:01.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:01 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/4079606771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:01.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:01 smithi195 ceph-mon[58095]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:01.467 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:01 smithi195 ceph-mon[58095]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-27T07:55:01.467 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:01 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/4079606771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:02.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:02 smithi149 ceph-mon[64590]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:02.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:02 smithi149 ceph-mon[67157]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:02 smithi195 ceph-mon[58095]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:03.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:03 smithi149 ceph-mon[67157]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-02-27T07:55:03.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:03 smithi149 ceph-mon[64590]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-02-27T07:55:03.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:03 smithi195 ceph-mon[58095]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-02-27T07:55:05.148 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:04] "GET /metrics HTTP/1.1" 200 38451 "" "Prometheus/2.33.4" 2024-02-27T07:55:05.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:05 smithi149 ceph-mon[67157]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:05.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:05 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:05.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:05 smithi149 ceph-mon[64590]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:05.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:05 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:05 smithi195 ceph-mon[58095]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:05 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:07.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:07 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:07.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:07 smithi149 ceph-mon[67157]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:07 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:07.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:07 smithi149 ceph-mon[64590]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:07 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:07.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:07 smithi195 ceph-mon[58095]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:08.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:55:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:55:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:55:09.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:09 smithi149 ceph-mon[67157]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:09 smithi149 ceph-mon[64590]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:09.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:09 smithi195 ceph-mon[58095]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:10 smithi195 ceph-mon[58095]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:10.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:10 smithi149 ceph-mon[64590]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:10.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:10 smithi149 ceph-mon[67157]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:11.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:11 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:11.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:11 smithi149 ceph-mon[64590]: Upgrade: osd.3 is safe to restart 2024-02-27T07:55:11.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:11 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:11.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:11 smithi149 ceph-mon[67157]: Upgrade: osd.3 is safe to restart 2024-02-27T07:55:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:11 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T07:55:11.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:11 smithi195 ceph-mon[58095]: Upgrade: osd.3 is safe to restart 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: Upgrade: Updating osd.3 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: Deploying daemon osd.3 on smithi149 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[64590]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: Upgrade: Updating osd.3 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:12.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:55:12.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:12.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: Deploying daemon osd.3 on smithi149 2024-02-27T07:55:12.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:12 smithi149 ceph-mon[67157]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: Upgrade: Updating osd.3 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: Deploying daemon osd.3 on smithi149 2024-02-27T07:55:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:12 smithi195 ceph-mon[58095]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:14.441 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:14 smithi149 ceph-mon[67157]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:14.442 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:14 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:14.442 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:14 smithi149 ceph-mon[64590]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:14.442 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:14 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:14.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:14 smithi195 ceph-mon[58095]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:14.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:14 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:14] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-02-27T07:55:15.177 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:15 smithi149 systemd[1]: Stopping Ceph osd.3 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:55:15.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:15 smithi149 ceph-mon[67157]: osd.3 marked itself down and dead 2024-02-27T07:55:15.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:15 smithi149 ceph-mon[64590]: osd.3 marked itself down and dead 2024-02-27T07:55:15.677 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[41253]: 2024-02-27T07:55:15.401+0000 7fa2f9012700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:55:15.677 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[41253]: 2024-02-27T07:55:15.401+0000 7fa2f9012700 -1 osd.3 120 *** Got signal Terminated *** 2024-02-27T07:55:15.677 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:15 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[41253]: 2024-02-27T07:55:15.401+0000 7fa2f9012700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:55:15.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:15 smithi195 ceph-mon[58095]: osd.3 marked itself down and dead 2024-02-27T07:55:16.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:15 smithi149 podman[77501]: 2024-02-27 07:55:15.916076279 +0000 UTC m=+0.748534495 container died ab481c2a7928420bf90a6dd35c641fb8198b1afe22aa16a9186f27ca94baa384 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, release=754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-type=git, vendor=Red Hat, Inc., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, version=8, name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream) 2024-02-27T07:55:16.477 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:16 smithi149 podman[77501]: 2024-02-27 07:55:16.277359138 +0000 UTC m=+1.109817355 container cleanup ab481c2a7928420bf90a6dd35c641fb8198b1afe22aa16a9186f27ca94baa384 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, vcs-type=git, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-27T07:55:16.477 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:16 smithi149 bash[77501]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3 2024-02-27T07:55:16.893 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:16.893 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:16.893 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[64590]: osdmap e121: 8 total, 7 up, 8 in 2024-02-27T07:55:16.893 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[64590]: pgmap v110: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:55:16.893 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[64590]: osdmap e122: 8 total, 7 up, 8 in 2024-02-27T07:55:16.894 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:16.894 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:16.894 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[67157]: osdmap e121: 8 total, 7 up, 8 in 2024-02-27T07:55:16.894 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[67157]: pgmap v110: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:55:16.894 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:16 smithi149 ceph-mon[67157]: osdmap e122: 8 total, 7 up, 8 in 2024-02-27T07:55:16.894 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:16 smithi149 podman[77518]: 2024-02-27 07:55:16.562042194 +0000 UTC m=+0.641235384 container remove ab481c2a7928420bf90a6dd35c641fb8198b1afe22aa16a9186f27ca94baa384 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, architecture=x86_64, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870) 2024-02-27T07:55:16.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:16 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:16.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:16 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:16.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:16 smithi195 ceph-mon[58095]: osdmap e121: 8 total, 7 up, 8 in 2024-02-27T07:55:16.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:16 smithi195 ceph-mon[58095]: pgmap v110: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:55:16.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:16 smithi195 ceph-mon[58095]: osdmap e122: 8 total, 7 up, 8 in 2024-02-27T07:55:17.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:16 smithi149 podman[77609]: 2024-02-27 07:55:16.796353119 +0000 UTC m=+0.018206622 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:17.177 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:16 smithi149 podman[77609]: 2024-02-27 07:55:16.913091815 +0000 UTC m=+0.134945311 container create e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-27T07:55:17.502 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:17 smithi149 podman[77609]: 2024-02-27 07:55:17.199112246 +0000 UTC m=+0.420965756 container init e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:55:17.503 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:17 smithi149 podman[77609]: 2024-02-27 07:55:17.202295338 +0000 UTC m=+0.424148844 container start e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:55:17.503 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:17 smithi149 podman[77609]: 2024-02-27 07:55:17.265674504 +0000 UTC m=+0.487528001 container attach e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-02-27T07:55:17.915 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:17 smithi149 podman[77609]: 2024-02-27 07:55:17.502534976 +0000 UTC m=+0.724388480 container died e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True) 2024-02-27T07:55:18.166 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77635]: 2024-02-27 07:55:18.152526045 +0000 UTC m=+0.643632350 container remove e36c53840a27e4f03cdfb69e265ddbe5b36a367c7c393f6932ea93c09b30a6da (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-27T07:55:18.426 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-27T07:55:18.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 systemd[1]: Stopped Ceph osd.3 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:55:18.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.3.service: Consumed 7.998s CPU time. 2024-02-27T07:55:18.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 systemd[1]: Starting Ceph osd.3 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:55:18.898 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77738]: 2024-02-27 07:55:18.504882292 +0000 UTC m=+0.018240256 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:18.898 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77738]: 2024-02-27 07:55:18.611688309 +0000 UTC m=+0.125046316 container create d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:55:18.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:55:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:55:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:55:19.160 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77738]: 2024-02-27 07:55:18.898088204 +0000 UTC m=+0.411446178 container init d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-02-27T07:55:19.160 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77738]: 2024-02-27 07:55:18.901616908 +0000 UTC m=+0.414974877 container start d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:55:19.160 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:18 smithi149 podman[77738]: 2024-02-27 07:55:18.98100741 +0000 UTC m=+0.494365386 container attach d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-27T07:55:19.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-mon[67157]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:55:19.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-mon[64590]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 bash[77738]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-27T07:55:19.427 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T07:55:19.428 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T07:55:19.428 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-27T07:55:19.428 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T07:55:19.428 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate[77753]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-27T07:55:19.428 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 podman[77738]: 2024-02-27 07:55:19.366132672 +0000 UTC m=+0.879490638 container died d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-27T07:55:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:19 smithi195 ceph-mon[58095]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:55:20.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:19 smithi149 podman[77874]: 2024-02-27 07:55:19.917754787 +0000 UTC m=+0.542232269 container remove d97e3e14971cca56e218e172fc4092915b66132a92e3bd2c4e02b3849ab509de (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-27T07:55:20.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 podman[77939]: 2024-02-27 07:55:20.032742395 +0000 UTC m=+0.017955650 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:20.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 podman[77939]: 2024-02-27 07:55:20.160277877 +0000 UTC m=+0.145491117 container create edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-27T07:55:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:20 smithi195 ceph-mon[58095]: pgmap v113: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 102 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-02-27T07:55:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:20 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-mon[67157]: pgmap v113: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 102 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-mon[64590]: pgmap v113: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 102 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 61/630 objects degraded (9.683%) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 podman[77939]: 2024-02-27 07:55:20.429926464 +0000 UTC m=+0.415139710 container init edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:55:20.538 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 podman[77939]: 2024-02-27 07:55:20.433281253 +0000 UTC m=+0.418494498 container start edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-27T07:55:20.539 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-osd[77958]: -- 172.21.15.149:0/3620943746 <== mon.0 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c72a2ef680 con 0x55c72a2b9400 2024-02-27T07:55:20.926 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 bash[77939]: edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d 2024-02-27T07:55:20.926 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 systemd[1]: Started Ceph osd.3 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:55:21.411 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:20 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T07:55:20.992+0000 7f378a4097c0 -1 Falling back to public interface 2024-02-27T07:55:21.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:21.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:21.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:21.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:21.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:22.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:22 smithi149 ceph-mon[67157]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:22.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:22 smithi149 ceph-mon[64590]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:22.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:22 smithi195 ceph-mon[58095]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:24.044 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:23 smithi149 ceph-mon[64590]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:55:24.045 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:23 smithi149 ceph-mon[67157]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:55:24.045 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T07:55:23.723+0000 7f378a4097c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-27T07:55:24.045 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T07:55:23.752+0000 7f378a4097c0 -1 osd.3 120 log_to_monitors true 2024-02-27T07:55:24.045 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 07:55:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T07:55:23.840+0000 7f377b861700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:55:24.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:23 smithi195 ceph-mon[58095]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T07:55:24.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[67157]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:55:24.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[67157]: osdmap e123: 8 total, 7 up, 8 in 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[67157]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[67157]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:24] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[64590]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[64590]: osdmap e123: 8 total, 7 up, 8 in 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[64590]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:55:24.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:24 smithi149 ceph-mon[64590]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:24 smithi195 ceph-mon[58095]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T07:55:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:24 smithi195 ceph-mon[58095]: osdmap e123: 8 total, 7 up, 8 in 2024-02-27T07:55:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:24 smithi195 ceph-mon[58095]: from='osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-02-27T07:55:25.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:24 smithi195 ceph-mon[58095]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 80/630 objects degraded (12.698%) 2024-02-27T07:55:26.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:55:26.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680] boot 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: osdmap e124: 8 total, 8 up, 8 in 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680] boot 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: osdmap e124: 8 total, 8 up, 8 in 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:26.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: osd.3 [v2:172.21.15.149:6826/3344025680,v1:172.21.15.149:6827/3344025680] boot 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: osdmap e124: 8 total, 8 up, 8 in 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:26.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[64590]: osdmap e125: 8 total, 8 up, 8 in 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[64590]: pgmap v119: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.120 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:27.121 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[67157]: osdmap e125: 8 total, 8 up, 8 in 2024-02-27T07:55:27.121 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[67157]: pgmap v119: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:27.121 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.121 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:26 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:26 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:26 smithi195 ceph-mon[58095]: osdmap e125: 8 total, 8 up, 8 in 2024-02-27T07:55:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:26 smithi195 ceph-mon[58095]: pgmap v119: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:26 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:27.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:26 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:28.134 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:27 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:28.134 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:27 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:28.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:27 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:28.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:28 smithi195 ceph-mon[58095]: pgmap v120: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:28.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:55:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:55:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:55:29.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:28 smithi149 ceph-mon[64590]: pgmap v120: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:29.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:28 smithi149 ceph-mon[67157]: pgmap v120: 161 pgs: 31 peering, 14 active+undersized+degraded, 21 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 49/630 objects degraded (7.778%) 2024-02-27T07:55:29.926 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:29 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:55:29.826+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:55:30.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.429 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:55:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:55:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:30.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:30.934 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:55:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:55:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[67157]: pgmap v121: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded) 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[64590]: pgmap v121: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded) 2024-02-27T07:55:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:31 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:55:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:31 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:31 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:55:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:31 smithi195 ceph-mon[58095]: pgmap v121: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:31 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/630 objects degraded (7.778%), 14 pgs degraded) 2024-02-27T07:55:31.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:31 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (4m) 6s ago 9m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (4m) 2m ago 9m 44.6M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (3m) 6s ago 8m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (4m) 6s ago 15m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (3m) 6s ago 16m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (2m) 2m ago 14m 17.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (3m) 6s ago 14m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:55:31.582 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (5m) 6s ago 10m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (4m) 2m ago 10m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (115s) 6s ago 13m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (80s) 6s ago 13m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (46s) 6s ago 13m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (11s) 6s ago 12m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (12m) 2m ago 12m 53.8M 2992M 17.2.0 e1d6a67b021e 43a2ba62bd39 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (12m) 2m ago 12m 49.9M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (11m) 2m ago 11m 49.1M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (11m) 2m ago 11m 52.0M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (3m) 2m ago 10m 55.6M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (9m) 6s ago 9m 72.6M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (8m) 2m ago 8m 72.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (8m) 6s ago 8m 74.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:55:31.583 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (8m) 2m ago 8m 73.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-27T07:55:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 9 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:55:31.978 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:55:32.313 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:55:32.313 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:55:32.313 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "9/23 daemons upgraded", 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:55:32.314 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:55:32.314 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:32 smithi149 ceph-mon[64590]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:32.314 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:32 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3446172506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:32.315 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:32 smithi149 ceph-mon[67157]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:32.315 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:32 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3446172506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:32 smithi195 ceph-mon[58095]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:32 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3446172506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:55:32.705 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:55:33.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[64590]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[64590]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[64590]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/4234377737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[67157]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[67157]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[67157]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:33.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:33 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/4234377737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:33 smithi195 ceph-mon[58095]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:33 smithi195 ceph-mon[58095]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:33 smithi195 ceph-mon[58095]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:55:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:33 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/4234377737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:55:34.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:34 smithi149 ceph-mon[67157]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:34 smithi149 ceph-mon[64590]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:34.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:34 smithi195 ceph-mon[58095]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:55:35.146 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:34] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-02-27T07:55:35.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:35 smithi149 ceph-mon[67157]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s 2024-02-27T07:55:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:35 smithi149 ceph-mon[64590]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s 2024-02-27T07:55:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:35 smithi195 ceph-mon[58095]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s 2024-02-27T07:55:37.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:37 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:37 smithi149 ceph-mon[67157]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 398 B/s rd, 0 op/s 2024-02-27T07:55:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:37 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:37 smithi149 ceph-mon[64590]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 398 B/s rd, 0 op/s 2024-02-27T07:55:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:37 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:37 smithi195 ceph-mon[58095]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 398 B/s rd, 0 op/s 2024-02-27T07:55:38.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 07:55:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T07:55:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.149\", device_class=\"ssd\", hostname=\"smithi149\", instance=\"172.21.15.195:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.149\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-27T07:55:39.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:39 smithi149 ceph-mon[67157]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T07:55:39.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:39 smithi149 ceph-mon[64590]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T07:55:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:39 smithi195 ceph-mon[58095]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T07:55:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:40 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:40 smithi195 ceph-mon[58095]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:55:40.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:40 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:40.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:40 smithi149 ceph-mon[67157]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:55:40.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:40 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:40.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:40 smithi149 ceph-mon[64590]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-27T07:55:42.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:42 smithi149 ceph-mon[67157]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:42 smithi149 ceph-mon[64590]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:42 smithi195 ceph-mon[58095]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:45.134 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:44] "GET /metrics HTTP/1.1" 200 38458 "" "Prometheus/2.33.4" 2024-02-27T07:55:45.421 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:45 smithi195 ceph-mon[58095]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:45.421 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:45.421 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:45.421 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:45.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[67157]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[64590]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:46.403 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:46 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:46.403 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:46 smithi195 ceph-mon[58095]: Upgrade: osd.4 is safe to restart 2024-02-27T07:55:46.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:46 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:46.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:46 smithi149 ceph-mon[67157]: Upgrade: osd.4 is safe to restart 2024-02-27T07:55:46.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:46 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T07:55:46.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:46 smithi149 ceph-mon[64590]: Upgrade: osd.4 is safe to restart 2024-02-27T07:55:47.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:47 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:47.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:47 smithi195 ceph-mon[58095]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:47.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:47 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:47.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:47 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:55:47.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:47 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:47.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:47.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[67157]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[64590]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T07:55:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:47 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:55:48.341 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:48 smithi195 ceph-mon[58095]: Upgrade: Updating osd.4 2024-02-27T07:55:48.341 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:48 smithi195 ceph-mon[58095]: Deploying daemon osd.4 on smithi195 2024-02-27T07:55:48.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:48 smithi149 ceph-mon[67157]: Upgrade: Updating osd.4 2024-02-27T07:55:48.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:48 smithi149 ceph-mon[67157]: Deploying daemon osd.4 on smithi195 2024-02-27T07:55:48.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:48 smithi149 ceph-mon[64590]: Upgrade: Updating osd.4 2024-02-27T07:55:48.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:48 smithi149 ceph-mon[64590]: Deploying daemon osd.4 on smithi195 2024-02-27T07:55:49.410 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:49 smithi195 ceph-mon[58095]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:49.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:49 smithi149 ceph-mon[64590]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:49.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:49 smithi149 ceph-mon[67157]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:55:50.059 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:49 smithi195 systemd[1]: Stopping Ceph osd.4 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:55:50.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:50 smithi149 ceph-mon[67157]: osd.4 marked itself down and dead 2024-02-27T07:55:50.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:50 smithi149 ceph-mon[64590]: osd.4 marked itself down and dead 2024-02-27T07:55:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:50 smithi195 ceph-mon[58095]: osd.4 marked itself down and dead 2024-02-27T07:55:50.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[29151]: 2024-02-27T07:55:50.055+0000 7f2545521700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:55:50.467 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[29151]: 2024-02-27T07:55:50.055+0000 7f2545521700 -1 osd.4 125 *** Got signal Terminated *** 2024-02-27T07:55:50.467 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:50 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[29151]: 2024-02-27T07:55:50.055+0000 7f2545521700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:55:50.823 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:50 smithi195 podman[60909]: 2024-02-27 07:55:50.553983734 +0000 UTC m=+0.604481028 container died 43a2ba62bd398910d84b5897213e535b61edf7f2f5fe7c6314067f79b1e3b292 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, version=8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, name=centos-stream, RELEASE=HEAD, architecture=x86_64, release=754) 2024-02-27T07:55:51.158 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[60909]: 2024-02-27 07:55:51.005735745 +0000 UTC m=+1.056233037 container cleanup 43a2ba62bd398910d84b5897213e535b61edf7f2f5fe7c6314067f79b1e3b292 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, architecture=x86_64) 2024-02-27T07:55:51.159 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 bash[60909]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4 2024-02-27T07:55:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[67157]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[67157]: osdmap e126: 8 total, 7 up, 8 in 2024-02-27T07:55:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[64590]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:51 smithi149 ceph-mon[64590]: osdmap e126: 8 total, 7 up, 8 in 2024-02-27T07:55:51.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[60938]: 2024-02-27 07:55:51.315677219 +0000 UTC m=+0.763503239 container remove 43a2ba62bd398910d84b5897213e535b61edf7f2f5fe7c6314067f79b1e3b292 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.component=centos-stream-container, version=8, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-27T07:55:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:51 smithi195 ceph-mon[58095]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:55:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:51 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:55:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:51 smithi195 ceph-mon[58095]: osdmap e126: 8 total, 7 up, 8 in 2024-02-27T07:55:51.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[61033]: 2024-02-27 07:55:51.567268166 +0000 UTC m=+0.017942217 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:51.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[61033]: 2024-02-27 07:55:51.691584785 +0000 UTC m=+0.142258820 container create 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-27T07:55:51.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[61033]: 2024-02-27 07:55:51.910935482 +0000 UTC m=+0.361609515 container init 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-27T07:55:51.965 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[61033]: 2024-02-27 07:55:51.913991534 +0000 UTC m=+0.364665577 container start 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-02-27T07:55:52.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:52 smithi149 ceph-mon[67157]: osdmap e127: 8 total, 7 up, 8 in 2024-02-27T07:55:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:52 smithi149 ceph-mon[64590]: osdmap e127: 8 total, 7 up, 8 in 2024-02-27T07:55:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:52 smithi195 ceph-mon[58095]: osdmap e127: 8 total, 7 up, 8 in 2024-02-27T07:55:52.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:51 smithi195 podman[61033]: 2024-02-27 07:55:51.994154696 +0000 UTC m=+0.444828743 container attach 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:55:52.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:52 smithi195 podman[61033]: 2024-02-27 07:55:52.204803645 +0000 UTC m=+0.655477679 container died 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-27T07:55:53.169 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:52 smithi195 podman[61059]: 2024-02-27 07:55:52.898666808 +0000 UTC m=+0.688315799 container remove 0dd39f11c431abf76b1105626a0e5989d9bb1de3b46dee39c08c5770b892ad22 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:55:53.170 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:52 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-27T07:55:53.170 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:52 smithi195 systemd[1]: Stopped Ceph osd.4 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:55:53.170 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:52 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.4.service: Consumed 8.632s CPU time. 2024-02-27T07:55:53.170 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 systemd[1]: Starting Ceph osd.4 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:55:53.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:53 smithi149 ceph-mon[67157]: pgmap v134: 161 pgs: 10 peering, 21 stale+active+clean, 130 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:55:53.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:53 smithi149 ceph-mon[67157]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-27T07:55:53.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:53 smithi149 ceph-mon[64590]: pgmap v134: 161 pgs: 10 peering, 21 stale+active+clean, 130 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:55:53.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:53 smithi149 ceph-mon[64590]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-27T07:55:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:53 smithi195 ceph-mon[58095]: pgmap v134: 161 pgs: 10 peering, 21 stale+active+clean, 130 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:55:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:53 smithi195 ceph-mon[58095]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-27T07:55:53.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 podman[61162]: 2024-02-27 07:55:53.259137711 +0000 UTC m=+0.017124154 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:53.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 podman[61162]: 2024-02-27 07:55:53.366679371 +0000 UTC m=+0.124665837 container create 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-02-27T07:55:53.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 podman[61162]: 2024-02-27 07:55:53.587744044 +0000 UTC m=+0.345730475 container init 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T07:55:53.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 podman[61162]: 2024-02-27 07:55:53.59169361 +0000 UTC m=+0.349680043 container start 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-27T07:55:53.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:53 smithi195 podman[61162]: 2024-02-27 07:55:53.654549223 +0000 UTC m=+0.412535656 container attach 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-27T07:55:54.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-mon[58095]: pgmap v135: 161 pgs: 2 active+undersized+degraded, 10 peering, 15 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 5/630 objects degraded (0.794%) 2024-02-27T07:55:54.408 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T07:55:54.408 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 bash[61162]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate[61178]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-27T07:55:54.409 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 podman[61162]: 2024-02-27 07:55:54.063709753 +0000 UTC m=+0.821696177 container died 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:55:54.579 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:54 smithi149 ceph-mon[67157]: pgmap v135: 161 pgs: 2 active+undersized+degraded, 10 peering, 15 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 5/630 objects degraded (0.794%) 2024-02-27T07:55:54.579 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:54 smithi149 ceph-mon[64590]: pgmap v135: 161 pgs: 2 active+undersized+degraded, 10 peering, 15 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 5/630 objects degraded (0.794%) 2024-02-27T07:55:54.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 podman[61295]: 2024-02-27 07:55:54.708383868 +0000 UTC m=+0.637936796 container remove 7cb47eb0b6109c7c4ab6be3fd404ed3d0a607d2211167bb57e986b6d22ac0ad6 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-27T07:55:54.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 podman[61358]: 2024-02-27 07:55:54.925964742 +0000 UTC m=+0.110992512 container create 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-27T07:55:54.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:54 smithi195 podman[61358]: 2024-02-27 07:55:54.83335289 +0000 UTC m=+0.018380631 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:55:55.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:55:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:55:54] "GET /metrics HTTP/1.1" 200 38458 "" "Prometheus/2.33.4" 2024-02-27T07:55:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:55 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 5/630 objects degraded (0.794%), 2 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:55.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 podman[61358]: 2024-02-27 07:55:55.153755599 +0000 UTC m=+0.338783326 container init 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:55:55.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 podman[61358]: 2024-02-27 07:55:55.156984628 +0000 UTC m=+0.342012355 container start 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:55:55.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 ceph-osd[61377]: -- 172.21.15.195:0/2058718824 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c933b23680 con 0x55c933aed000 2024-02-27T07:55:55.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 bash[61358]: 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d 2024-02-27T07:55:55.465 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 systemd[1]: Started Ceph osd.4 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:55:55.507 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:55 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 5/630 objects degraded (0.794%), 2 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:55.507 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:55 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 5/630 objects degraded (0.794%), 2 pgs degraded (PG_DEGRADED) 2024-02-27T07:55:55.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:55 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T07:55:55.702+0000 7f3486f077c0 -1 Falling back to public interface 2024-02-27T07:55:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:56 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:56 smithi195 ceph-mon[58095]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:56.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[67157]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:56.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:56 smithi149 ceph-mon[64590]: pgmap v136: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:57.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:57 smithi149 ceph-mon[67157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-27T07:55:57.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:57 smithi149 ceph-mon[64590]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-27T07:55:57.704 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:57 smithi195 ceph-mon[58095]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-27T07:55:58.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:58 smithi149 ceph-mon[67157]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:58.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:58 smithi149 ceph-mon[64590]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:58.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:58 smithi195 ceph-mon[58095]: pgmap v137: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:55:58.714 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T07:55:58.415+0000 7f3486f077c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-27T07:55:58.714 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T07:55:58.449+0000 7f3486f077c0 -1 osd.4 125 log_to_monitors true 2024-02-27T07:55:59.606 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:59 smithi195 ceph-mon[58095]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.607 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:59 smithi195 ceph-mon[58095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.607 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:59.607 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:55:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[67157]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[67157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[64590]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[64590]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:55:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:55:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:55:59.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 07:55:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T07:55:59.605+0000 7f3474357700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:56:00.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: osdmap e128: 8 total, 7 up, 8 in 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: from='osd.4 ' entity='osd.4' 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[67157]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: osdmap e128: 8 total, 7 up, 8 in 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: from='osd.4 ' entity='osd.4' 2024-02-27T07:56:00.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:00 smithi149 ceph-mon[64590]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: osdmap e128: 8 total, 7 up, 8 in 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: from='osd.4 ' entity='osd.4' 2024-02-27T07:56:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:00 smithi195 ceph-mon[58095]: pgmap v139: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%) 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308] boot 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: osdmap e129: 8 total, 8 up, 8 in 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:01.568 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:01 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:01.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:01.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308] boot 2024-02-27T07:56:01.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: osdmap e129: 8 total, 8 up, 8 in 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: osd.4 [v2:172.21.15.195:6800/1635194308,v1:172.21.15.195:6801/1635194308] boot 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: osdmap e129: 8 total, 8 up, 8 in 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:01.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:01 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:02.867 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:02 smithi195 ceph-mon[58095]: osdmap e130: 8 total, 8 up, 8 in 2024-02-27T07:56:02.867 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:02 smithi195 ceph-mon[58095]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%) 2024-02-27T07:56:02.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:02 smithi149 ceph-mon[64590]: osdmap e130: 8 total, 8 up, 8 in 2024-02-27T07:56:02.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:02 smithi149 ceph-mon[64590]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%) 2024-02-27T07:56:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:02 smithi149 ceph-mon[67157]: osdmap e130: 8 total, 8 up, 8 in 2024-02-27T07:56:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:02 smithi149 ceph-mon[67157]: pgmap v142: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%) 2024-02-27T07:56:03.054 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:56:03.710 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:56:03.710 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (5m) 38s ago 10m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:56:03.710 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (4m) 4s ago 10m 53.9M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:56:03.710 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (4m) 38s ago 9m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (4m) 4s ago 14m 450M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (4m) 38s ago 16m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (4m) 38s ago 16m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (3m) 4s ago 14m 35.3M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (3m) 38s ago 15m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (5m) 38s ago 11m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (5m) 4s ago 10m 17.6M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (2m) 38s ago 14m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (113s) 38s ago 14m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:56:03.711 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (78s) 38s ago 13m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (43s) 38s ago 13m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (8s) 4s ago 12m 11.7M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (12m) 4s ago 12m 55.8M 2992M 17.2.0 e1d6a67b021e 94c67860988d 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (12m) 4s ago 12m 53.3M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (11m) 4s ago 11m 57.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (4m) 4s ago 11m 68.2M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (9m) 38s ago 9m 72.6M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (9m) 4s ago 9m 72.7M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (9m) 38s ago 9m 74.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:56:03.712 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (9m) 4s ago 9m 74.5M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:56:03.713 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:03 smithi195 ceph-mon[58095]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:03.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:03 smithi149 ceph-mon[64590]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:03.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:03 smithi149 ceph-mon[67157]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.113 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:56:04.114 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:56:04.114 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:56:04.114 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:04.114 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:56:04.114 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:04.115 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 10 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:56:04.116 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:56:04.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:56:04.356+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-27T07:56:04.452 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "10/23 daemons upgraded", 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:56:04.453 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2588079020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 38 active+undersized, 96 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.833 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:04.834 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2588079020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 38 active+undersized, 96 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.835 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:04.849 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN Degraded data redundancy: 104/630 objects degraded (16.508%), 27 pgs degraded 2024-02-27T07:56:04.849 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 104/630 objects degraded (16.508%), 27 pgs degraded 2024-02-27T07:56:04.849 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-27T07:56:04.850 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 5.18 is active+undersized+degraded, acting [6,1] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 5.1d is active+undersized+degraded, acting [1,0] 2024-02-27T07:56:04.851 INFO:teuthology.orchestra.run.smithi149.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-02-27T07:56:04.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:04.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:04.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2588079020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 38 active+undersized, 96 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:04.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:05.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:04] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-02-27T07:56:05.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:05.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[67157]: from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/34498100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[64590]: from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:05.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:05 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/34498100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:05.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:05 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:05.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:05 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-02-27T07:56:05.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:05 smithi195 ceph-mon[58095]: from='client.44424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:05.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:05 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/34498100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:06.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:06 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:06.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:06 smithi149 ceph-mon[67157]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-02-27T07:56:06.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:06 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:06.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:06 smithi149 ceph-mon[64590]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-02-27T07:56:06.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:06 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:06.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:06 smithi195 ceph-mon[58095]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-02-27T07:56:07.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:07 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/630 objects degraded (16.508%), 27 pgs degraded) 2024-02-27T07:56:07.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:07 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:56:07.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:07 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/630 objects degraded (16.508%), 27 pgs degraded) 2024-02-27T07:56:07.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:07 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:56:07.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:07 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/630 objects degraded (16.508%), 27 pgs degraded) 2024-02-27T07:56:07.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:07 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:56:08.884 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:08 smithi195 ceph-mon[58095]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:08.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:08 smithi149 ceph-mon[67157]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:08.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:08 smithi149 ceph-mon[64590]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:10 smithi195 ceph-mon[58095]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:56:10.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:10 smithi149 ceph-mon[67157]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:56:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:10 smithi149 ceph-mon[64590]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:56:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:12 smithi149 ceph-mon[67157]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-27T07:56:12.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:12 smithi149 ceph-mon[64590]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-27T07:56:12.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:12 smithi195 ceph-mon[58095]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-27T07:56:15.135 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:14] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-02-27T07:56:15.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[64590]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[67157]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:15.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:15 smithi195 ceph-mon[58095]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:17 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:17 smithi149 ceph-mon[64590]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:17 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:17 smithi149 ceph-mon[67157]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:17 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:17 smithi195 ceph-mon[58095]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:19.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:19 smithi149 ceph-mon[67157]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:19.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:19 smithi149 ceph-mon[64590]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:19 smithi195 ceph-mon[58095]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:20.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:20 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:20.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:20 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:20.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:20 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:21.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: Upgrade: osd.5 is safe to restart 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: Upgrade: osd.5 is safe to restart 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:56:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:21 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: Upgrade: osd.5 is safe to restart 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T07:56:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:21 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:22.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:22 smithi149 ceph-mon[64590]: Upgrade: Updating osd.5 2024-02-27T07:56:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:22 smithi149 ceph-mon[64590]: Deploying daemon osd.5 on smithi195 2024-02-27T07:56:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:22 smithi149 ceph-mon[67157]: Upgrade: Updating osd.5 2024-02-27T07:56:22.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:22 smithi149 ceph-mon[67157]: Deploying daemon osd.5 on smithi195 2024-02-27T07:56:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:22 smithi195 ceph-mon[58095]: Upgrade: Updating osd.5 2024-02-27T07:56:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:22 smithi195 ceph-mon[58095]: Deploying daemon osd.5 on smithi195 2024-02-27T07:56:23.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:23 smithi195 ceph-mon[58095]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:23.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:23 smithi149 ceph-mon[64590]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:23.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:23 smithi149 ceph-mon[67157]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:56:24.614 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:24 smithi195 systemd[1]: Stopping Ceph osd.5 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:56:24.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[32031]: 2024-02-27T07:56:24.612+0000 7f4309324700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:56:24.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[32031]: 2024-02-27T07:56:24.612+0000 7f4309324700 -1 osd.5 130 *** Got signal Terminated *** 2024-02-27T07:56:24.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[32031]: 2024-02-27T07:56:24.612+0000 7f4309324700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:56:25.158 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:24] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-02-27T07:56:25.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:25 smithi149 ceph-mon[67157]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:25.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:25 smithi149 ceph-mon[67157]: osd.5 marked itself down and dead 2024-02-27T07:56:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:25 smithi149 ceph-mon[64590]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:25 smithi149 ceph-mon[64590]: osd.5 marked itself down and dead 2024-02-27T07:56:25.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:25 smithi195 ceph-mon[58095]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:56:25.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:25 smithi195 ceph-mon[58095]: osd.5 marked itself down and dead 2024-02-27T07:56:25.821 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:25 smithi195 podman[63262]: 2024-02-27 07:56:25.501037298 +0000 UTC m=+1.000103725 container died 94c67860988d9adfecba81e5a0e8337da3fc4dabd60d7b2f22192c21f8926cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, RELEASE=HEAD, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-27T07:56:25.821 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:25 smithi195 podman[63262]: 2024-02-27 07:56:25.821128869 +0000 UTC m=+1.320195293 container cleanup 94c67860988d9adfecba81e5a0e8337da3fc4dabd60d7b2f22192c21f8926cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, version=8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container) 2024-02-27T07:56:26.072 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:25 smithi195 bash[63262]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5 2024-02-27T07:56:26.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:26 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:56:26.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:26 smithi149 ceph-mon[67157]: osdmap e131: 8 total, 7 up, 8 in 2024-02-27T07:56:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:26 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:56:26.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:26 smithi149 ceph-mon[64590]: osdmap e131: 8 total, 7 up, 8 in 2024-02-27T07:56:26.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:26 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:56:26.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:26 smithi195 ceph-mon[58095]: osdmap e131: 8 total, 7 up, 8 in 2024-02-27T07:56:26.464 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63280]: 2024-02-27 07:56:26.072436124 +0000 UTC m=+0.574752431 container remove 94c67860988d9adfecba81e5a0e8337da3fc4dabd60d7b2f22192c21f8926cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, ceph=True, GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-27T07:56:26.793 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63371]: 2024-02-27 07:56:26.415163344 +0000 UTC m=+0.020748946 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:56:26.794 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63371]: 2024-02-27 07:56:26.548694392 +0000 UTC m=+0.154279973 container create aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:56:26.794 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63371]: 2024-02-27 07:56:26.793355365 +0000 UTC m=+0.398940952 container init aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-27T07:56:27.087 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63371]: 2024-02-27 07:56:26.796509786 +0000 UTC m=+0.402095373 container start aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:56:27.087 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:26 smithi195 podman[63371]: 2024-02-27 07:56:26.868327082 +0000 UTC m=+0.473912677 container attach aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0) 2024-02-27T07:56:27.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:27 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:27.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:27 smithi195 ceph-mon[58095]: pgmap v155: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:56:27.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:27 smithi195 ceph-mon[58095]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-27T07:56:27.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:27 smithi195 ceph-mon[58095]: osdmap e132: 8 total, 7 up, 8 in 2024-02-27T07:56:27.389 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 podman[63371]: 2024-02-27 07:56:27.087430809 +0000 UTC m=+0.693016393 container died aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:56:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[67157]: pgmap v155: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:56:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[67157]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-27T07:56:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[67157]: osdmap e132: 8 total, 7 up, 8 in 2024-02-27T07:56:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[64590]: pgmap v155: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:56:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[64590]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-27T07:56:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:27 smithi149 ceph-mon[64590]: osdmap e132: 8 total, 7 up, 8 in 2024-02-27T07:56:27.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 podman[63396]: 2024-02-27 07:56:27.680252239 +0000 UTC m=+0.585196334 container remove aefa617b29c1b8c5ac401b9972733fa06079c0e687a1d037321eb41bc5a03575 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-27T07:56:27.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-27T07:56:27.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 systemd[1]: Stopped Ceph osd.5 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:56:27.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.5.service: Consumed 8.837s CPU time. 2024-02-27T07:56:27.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:27 smithi195 systemd[1]: Starting Ceph osd.5 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:56:28.375 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.011812121 +0000 UTC m=+0.017587391 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:56:28.376 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.122914528 +0000 UTC m=+0.128689834 container create 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True) 2024-02-27T07:56:28.376 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.375544391 +0000 UTC m=+0.381319662 container init 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-27T07:56:28.714 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.378784972 +0000 UTC m=+0.384560241 container start 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:56:28.714 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.475948639 +0000 UTC m=+0.481723916 container attach 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T07:56:29.129 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 bash[63496]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate[63511]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-27T07:56:29.130 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:28 smithi195 podman[63496]: 2024-02-27 07:56:28.838275064 +0000 UTC m=+0.844050333 container died 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-27T07:56:29.404 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:29 smithi195 ceph-mon[58095]: pgmap v157: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:29.404 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:29.405 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:29.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[64590]: pgmap v157: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:29.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:29.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[67157]: pgmap v157: 161 pgs: 13 peering, 17 stale+active+clean, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:56:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:29.664 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 podman[63629]: 2024-02-27 07:56:29.404556826 +0000 UTC m=+0.561340872 container remove 436b2091b79ce55eb0aa4e68a1622fd0c4e8164277a6c712ffb5d63f8461bdaa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) 2024-02-27T07:56:29.664 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 podman[63695]: 2024-02-27 07:56:29.521169911 +0000 UTC m=+0.017698873 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:56:29.664 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 podman[63695]: 2024-02-27 07:56:29.664020918 +0000 UTC m=+0.160549871 container create 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-02-27T07:56:29.923 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 podman[63695]: 2024-02-27 07:56:29.899706108 +0000 UTC m=+0.396235061 container init 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:56:29.923 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 podman[63695]: 2024-02-27 07:56:29.903006131 +0000 UTC m=+0.399535091 container start 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-27T07:56:30.214 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:29 smithi195 ceph-osd[63713]: -- 172.21.15.195:0/2846292726 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d0c7dd680 con 0x563d0c7a7400 2024-02-27T07:56:30.214 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:30 smithi195 bash[63695]: 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc 2024-02-27T07:56:30.215 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:30 smithi195 systemd[1]: Started Ceph osd.5 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:56:30.714 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T07:56:30.447+0000 7f5deb5977c0 -1 Falling back to public interface 2024-02-27T07:56:31.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:31 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:31 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:31 smithi195 ceph-mon[58095]: pgmap v158: 161 pgs: 19 active+undersized+degraded, 13 peering, 28 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 75 op/s; 61/630 objects degraded (9.683%) 2024-02-27T07:56:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[67157]: pgmap v158: 161 pgs: 19 active+undersized+degraded, 13 peering, 28 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 75 op/s; 61/630 objects degraded (9.683%) 2024-02-27T07:56:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:31.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:31 smithi149 ceph-mon[64590]: pgmap v158: 161 pgs: 19 active+undersized+degraded, 13 peering, 28 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 75 op/s; 61/630 objects degraded (9.683%) 2024-02-27T07:56:32.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:32 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:32.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:32 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:32 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 61/630 objects degraded (9.683%), 19 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:33.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:33 smithi149 ceph-mon[67157]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:33.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:33 smithi149 ceph-mon[64590]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:33.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:33 smithi195 ceph-mon[58095]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:33.464 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T07:56:33.141+0000 7f5deb5977c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-27T07:56:33.464 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T07:56:33.169+0000 7f5deb5977c0 -1 osd.5 130 log_to_monitors true 2024-02-27T07:56:34.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:34 smithi195 ceph-mon[58095]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-27T07:56:34.355 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:34 smithi195 ceph-mon[58095]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.356 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:34 smithi195 ceph-mon[58095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[67157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-27T07:56:34.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[67157]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[67157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[64590]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-02-27T07:56:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[64590]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-mon[64590]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T07:56:34.714 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 07:56:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T07:56:34.354+0000 7f5dd89e7700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:56:35.085 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:34] "GET /metrics HTTP/1.1" 200 38536 "" "Prometheus/2.33.4" 2024-02-27T07:56:35.207 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:56:35.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:56:35.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: osdmap e133: 8 total, 7 up, 8 in 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 108 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[64590]: from='osd.5 ' entity='osd.5' 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: osdmap e133: 8 total, 7 up, 8 in 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 108 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:35.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:35 smithi149 ceph-mon[67157]: from='osd.5 ' entity='osd.5' 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: osdmap e133: 8 total, 7 up, 8 in 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 108 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:35.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:35 smithi195 ceph-mon[58095]: from='osd.5 ' entity='osd.5' 2024-02-27T07:56:35.874 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:56:35.874 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (6m) 70s ago 10m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:56:35.874 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (5m) 1s ago 10m 52.4M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:56:35.874 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (4m) 70s ago 9m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (4m) 1s ago 15m 450M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (5m) 70s ago 16m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (4m) 70s ago 17m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (3m) 1s ago 15m 36.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (4m) 70s ago 15m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (6m) 70s ago 11m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (5m) 1s ago 11m 17.8M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (3m) 70s ago 14m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (2m) 70s ago 14m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (110s) 70s ago 14m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (75s) 70s ago 13m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (41s) 1s ago 13m 51.1M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (6s) 1s ago 13m 14.8M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:56:35.875 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (12m) 1s ago 12m 55.3M 2992M 17.2.0 e1d6a67b021e 49040ee745e0 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (12m) 1s ago 12m 59.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (4m) 1s ago 11m 68.8M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (10m) 70s ago 10m 72.6M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (10m) 1s ago 10m 72.9M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (9m) 70s ago 9m 74.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:56:35.876 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (9m) 1s ago 9m 74.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:56:36.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:36.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805] boot 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: osdmap e134: 8 total, 8 up, 8 in 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805] boot 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: osdmap e134: 8 total, 8 up, 8 in 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:36 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: osd.5 [v2:172.21.15.195:6808/3821229805,v1:172.21.15.195:6809/3821229805] boot 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: osdmap e134: 8 total, 8 up, 8 in 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:36 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 6 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:56:36.269 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 11 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:56:36.270 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "11/23 daemons upgraded", 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:56:36.613 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:56:36.614 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN Degraded data redundancy: 71/630 objects degraded (11.270%), 20 pgs degraded 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 71/630 objects degraded (11.270%), 20 pgs degraded 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-02-27T07:56:37.005 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-02-27T07:56:37.006 INFO:teuthology.orchestra.run.smithi149.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-02-27T07:56:37.007 INFO:teuthology.orchestra.run.smithi149.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-27T07:56:37.286 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.286 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.44445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 175 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2730548683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: osdmap e135: 8 total, 8 up, 8 in 2024-02-27T07:56:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:37 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2646061905' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:37.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:37.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.44445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 175 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2730548683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: osdmap e135: 8 total, 8 up, 8 in 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2646061905' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.44445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 175 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2730548683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: osdmap e135: 8 total, 8 up, 8 in 2024-02-27T07:56:37.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:37 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2646061905' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:56:38.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:38 smithi195 ceph-mon[58095]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:38.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:38 smithi149 ceph-mon[64590]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:38.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:38 smithi149 ceph-mon[67157]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:56:39.426 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:56:39.061+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.428 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:39.429 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:39 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 40 active+undersized, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/630 objects degraded (11.270%) 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:56:39.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:39.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:39 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:56:40.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:40 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:40.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:40 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:56:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:40 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:40 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:56:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:40 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:40 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-02-27T07:56:41.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:41 smithi149 ceph-mon[67157]: pgmap v166: 161 pgs: 1 active+undersized+degraded, 7 active+undersized, 153 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 10/630 objects degraded (1.587%); 0 B/s, 7 objects/s recovering 2024-02-27T07:56:41.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:41 smithi149 ceph-mon[64590]: pgmap v166: 161 pgs: 1 active+undersized+degraded, 7 active+undersized, 153 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 10/630 objects degraded (1.587%); 0 B/s, 7 objects/s recovering 2024-02-27T07:56:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:41 smithi195 ceph-mon[58095]: pgmap v166: 161 pgs: 1 active+undersized+degraded, 7 active+undersized, 153 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 10/630 objects degraded (1.587%); 0 B/s, 7 objects/s recovering 2024-02-27T07:56:42.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[67157]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s; 0 B/s, 6 objects/s recovering 2024-02-27T07:56:42.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/630 objects degraded (1.587%), 1 pg degraded) 2024-02-27T07:56:42.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:56:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[64590]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s; 0 B/s, 6 objects/s recovering 2024-02-27T07:56:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/630 objects degraded (1.587%), 1 pg degraded) 2024-02-27T07:56:42.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:42 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:56:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:42 smithi195 ceph-mon[58095]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s; 0 B/s, 6 objects/s recovering 2024-02-27T07:56:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:42 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/630 objects degraded (1.587%), 1 pg degraded) 2024-02-27T07:56:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:42 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:56:45.136 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:44] "GET /metrics HTTP/1.1" 200 38543 "" "Prometheus/2.33.4" 2024-02-27T07:56:45.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[67157]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s; 0 B/s, 5 objects/s recovering 2024-02-27T07:56:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[64590]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s; 0 B/s, 5 objects/s recovering 2024-02-27T07:56:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:45 smithi195 ceph-mon[58095]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s; 0 B/s, 5 objects/s recovering 2024-02-27T07:56:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:47.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:47 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:47.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:47 smithi149 ceph-mon[67157]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:47 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:47.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:47 smithi149 ceph-mon[64590]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:47 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:47 smithi195 ceph-mon[58095]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:49.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:49 smithi149 ceph-mon[67157]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:49.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:49 smithi149 ceph-mon[64590]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:49 smithi195 ceph-mon[58095]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:50 smithi195 ceph-mon[58095]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:50.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:50.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:50 smithi149 ceph-mon[67157]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:50.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:50 smithi149 ceph-mon[64590]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 4 objects/s recovering 2024-02-27T07:56:52.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:52 smithi149 ceph-mon[64590]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 0 objects/s recovering 2024-02-27T07:56:52.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:52 smithi149 ceph-mon[67157]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 0 objects/s recovering 2024-02-27T07:56:52.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:52 smithi195 ceph-mon[58095]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 0 objects/s recovering 2024-02-27T07:56:54.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:54 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:54.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:54 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:54 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:55.134 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:56:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:56:54] "GET /metrics HTTP/1.1" 200 38543 "" "Prometheus/2.33.4" 2024-02-27T07:56:55.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:55.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[67157]: Upgrade: osd.6 is safe to restart 2024-02-27T07:56:55.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[67157]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-27T07:56:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[64590]: Upgrade: osd.6 is safe to restart 2024-02-27T07:56:55.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:55 smithi149 ceph-mon[64590]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-27T07:56:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:55 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T07:56:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:55 smithi195 ceph-mon[58095]: Upgrade: osd.6 is safe to restart 2024-02-27T07:56:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:55 smithi195 ceph-mon[58095]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: Upgrade: Updating osd.6 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: Deploying daemon osd.6 on smithi195 2024-02-27T07:56:57.097 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:56 smithi195 ceph-mon[58095]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-02-27T07:56:57.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:57.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: Upgrade: Updating osd.6 2024-02-27T07:56:57.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: Deploying daemon osd.6 on smithi195 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[67157]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: Upgrade: Updating osd.6 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: Deploying daemon osd.6 on smithi195 2024-02-27T07:56:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:56 smithi149 ceph-mon[64590]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-02-27T07:56:58.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:58 smithi195 ceph-mon[58095]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-27T07:56:58.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:58 smithi149 ceph-mon[64590]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-27T07:56:58.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:58 smithi149 ceph-mon[67157]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-27T07:56:59.544 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:56:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:59.545 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:56:59 smithi195 systemd[1]: Stopping Ceph osd.6 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:56:59.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:56:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:56:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:56:59.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:56:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[34912]: 2024-02-27T07:56:59.542+0000 7f789ddce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:56:59.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:56:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[34912]: 2024-02-27T07:56:59.542+0000 7f789ddce700 -1 osd.6 135 *** Got signal Terminated *** 2024-02-27T07:56:59.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:56:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[34912]: 2024-02-27T07:56:59.542+0000 7f789ddce700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:57:00.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:00 smithi149 ceph-mon[67157]: osd.6 marked itself down and dead 2024-02-27T07:57:00.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:00 smithi149 ceph-mon[67157]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-27T07:57:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:00 smithi149 ceph-mon[64590]: osd.6 marked itself down and dead 2024-02-27T07:57:00.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:00 smithi149 ceph-mon[64590]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-27T07:57:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:00 smithi195 ceph-mon[58095]: osd.6 marked itself down and dead 2024-02-27T07:57:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:00 smithi195 ceph-mon[58095]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-27T07:57:01.109 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:00 smithi195 podman[65598]: 2024-02-27 07:57:00.770672906 +0000 UTC m=+1.359783874 container died 49040ee745e0a77df8413117e9a03f7dbd6f31cf7636892306a43079a2c44d9c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.display-name=CentOS Stream 8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public) 2024-02-27T07:57:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:01 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:01 smithi195 ceph-mon[58095]: osdmap e136: 8 total, 7 up, 8 in 2024-02-27T07:57:01.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:01 smithi195 podman[65598]: 2024-02-27 07:57:01.10939779 +0000 UTC m=+1.698508755 container cleanup 49040ee745e0a77df8413117e9a03f7dbd6f31cf7636892306a43079a2c44d9c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, architecture=x86_64, release=754, vcs-type=git) 2024-02-27T07:57:01.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:01 smithi195 bash[65598]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6 2024-02-27T07:57:01.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:01 smithi195 podman[65616]: 2024-02-27 07:57:01.335763232 +0000 UTC m=+0.558799194 container remove 49040ee745e0a77df8413117e9a03f7dbd6f31cf7636892306a43079a2c44d9c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-type=git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., GIT_CLEAN=True) 2024-02-27T07:57:01.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:01 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:01.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:01 smithi149 ceph-mon[64590]: osdmap e136: 8 total, 7 up, 8 in 2024-02-27T07:57:01.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:01 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:01.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:01 smithi149 ceph-mon[67157]: osdmap e136: 8 total, 7 up, 8 in 2024-02-27T07:57:02.034 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:01 smithi195 podman[65707]: 2024-02-27 07:57:01.686128788 +0000 UTC m=+0.019143203 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:02.034 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:01 smithi195 podman[65707]: 2024-02-27 07:57:01.803390726 +0000 UTC m=+0.136405189 container create c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:57:02.034 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 podman[65707]: 2024-02-27 07:57:02.030598465 +0000 UTC m=+0.363612874 container init c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-27T07:57:02.034 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 podman[65707]: 2024-02-27 07:57:02.033738548 +0000 UTC m=+0.366752954 container start c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:57:02.319 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 podman[65707]: 2024-02-27 07:57:02.122378315 +0000 UTC m=+0.455392735 container attach c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-27T07:57:02.319 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:02 smithi195 ceph-mon[58095]: osdmap e137: 8 total, 7 up, 8 in 2024-02-27T07:57:02.319 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:02 smithi195 ceph-mon[58095]: pgmap v179: 161 pgs: 4 peering, 12 stale+active+clean, 145 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:57:02.676 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 podman[65707]: 2024-02-27 07:57:02.319237758 +0000 UTC m=+0.652252176 container died c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:57:02.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:02 smithi149 ceph-mon[64590]: osdmap e137: 8 total, 7 up, 8 in 2024-02-27T07:57:02.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:02 smithi149 ceph-mon[64590]: pgmap v179: 161 pgs: 4 peering, 12 stale+active+clean, 145 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:57:02.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:02 smithi149 ceph-mon[67157]: osdmap e137: 8 total, 7 up, 8 in 2024-02-27T07:57:02.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:02 smithi149 ceph-mon[67157]: pgmap v179: 161 pgs: 4 peering, 12 stale+active+clean, 145 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-27T07:57:03.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 podman[65733]: 2024-02-27 07:57:02.951278673 +0000 UTC m=+0.626468797 container remove c81a3964814bd6edc210b169646650e11939f35a3e886caac4a9de9c0e102b63 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:57:03.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-27T07:57:03.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 systemd[1]: Stopped Ceph osd.6 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:57:03.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:02 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6.service: Consumed 8.214s CPU time. 2024-02-27T07:57:03.215 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 systemd[1]: Starting Ceph osd.6 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:57:03.519 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:03 smithi195 ceph-mon[58095]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:03.519 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 podman[65833]: 2024-02-27 07:57:03.38543781 +0000 UTC m=+0.114506775 container create c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:57:03.520 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 podman[65833]: 2024-02-27 07:57:03.288882211 +0000 UTC m=+0.017951192 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:03.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:03 smithi149 ceph-mon[64590]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:03.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:03 smithi149 ceph-mon[67157]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:03.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 podman[65833]: 2024-02-27 07:57:03.621593605 +0000 UTC m=+0.350662585 container init c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-02-27T07:57:03.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 podman[65833]: 2024-02-27 07:57:03.624911263 +0000 UTC m=+0.353980230 container start c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:57:03.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:03 smithi195 podman[65833]: 2024-02-27 07:57:03.688165211 +0000 UTC m=+0.417234187 container attach c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T07:57:04.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-mon[58095]: pgmap v180: 161 pgs: 8 active+undersized+degraded, 4 peering, 9 stale+active+clean, 13 active+undersized, 127 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2024-02-27T07:57:04.400 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 bash[65833]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T07:57:04.401 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate[65848]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-27T07:57:04.402 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 podman[65833]: 2024-02-27 07:57:04.098155008 +0000 UTC m=+0.827223976 container died c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-27T07:57:04.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:04 smithi149 ceph-mon[67157]: pgmap v180: 161 pgs: 8 active+undersized+degraded, 4 peering, 9 stale+active+clean, 13 active+undersized, 127 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2024-02-27T07:57:04.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:04 smithi149 ceph-mon[64590]: pgmap v180: 161 pgs: 8 active+undersized+degraded, 4 peering, 9 stale+active+clean, 13 active+undersized, 127 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2024-02-27T07:57:04.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 podman[65966]: 2024-02-27 07:57:04.700540684 +0000 UTC m=+0.596926402 container remove c088e293f7b7b9ebc6bc9572a93ad033b96ce751feaca94b8c9ae12c6d020e25 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:57:04.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 podman[66031]: 2024-02-27 07:57:04.816368445 +0000 UTC m=+0.017689182 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:04.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:04 smithi195 podman[66031]: 2024-02-27 07:57:04.917970638 +0000 UTC m=+0.119291360 container create d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-27T07:57:05.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:04] "GET /metrics HTTP/1.1" 200 38541 "" "Prometheus/2.33.4" 2024-02-27T07:57:05.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:05 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 8 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:05.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 podman[66031]: 2024-02-27 07:57:05.112028832 +0000 UTC m=+0.313349554 container init d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS) 2024-02-27T07:57:05.465 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 podman[66031]: 2024-02-27 07:57:05.115007094 +0000 UTC m=+0.316327827 container start d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-27T07:57:05.465 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 ceph-osd[66051]: -- 172.21.15.195:0/1170411205 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f2d4ef680 con 0x557f2d4b8c00 2024-02-27T07:57:05.465 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 bash[66031]: d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 2024-02-27T07:57:05.465 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 systemd[1]: Started Ceph osd.6 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:57:05.654 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:05 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 8 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:05.654 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:05 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 8 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:05.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T07:57:05.658+0000 7f7e03c177c0 -1 Falling back to public interface 2024-02-27T07:57:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:06 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:06 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:06 smithi195 ceph-mon[58095]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:06.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[67157]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:06 smithi149 ceph-mon[64590]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:07.360 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:57:07.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:07 smithi195 ceph-mon[58095]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-27T07:57:07.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:07 smithi149 ceph-mon[67157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-27T07:57:07.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:07 smithi149 ceph-mon[64590]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (6m) 102s ago 11m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (5m) 33s ago 11m 52.4M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (5m) 102s ago 10m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (5m) 33s ago 15m 450M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:57:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (5m) 102s ago 17m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (5m) 102s ago 17m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (4m) 33s ago 16m 36.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (4m) 102s ago 16m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (6m) 102s ago 12m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (6m) 33s ago 12m 17.8M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (3m) 102s ago 15m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (2m) 102s ago 15m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (2m) 102s ago 14m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (107s) 102s ago 14m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (73s) 33s ago 14m 51.1M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (38s) 33s ago 13m 14.8M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 starting - - - 2992M 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (12m) 33s ago 12m 59.7M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (5m) 33s ago 12m 68.8M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (10m) 102s ago 10m 72.6M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:57:08.004 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (10m) 33s ago 10m 72.9M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:57:08.005 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (10m) 102s ago 10m 74.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:57:08.005 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (10m) 33s ago 10m 74.7M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:57:08.401 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 6 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 11 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:57:08.403 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:57:08.458 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-mon[58095]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-mon[58095]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-mon[58095]: from='client.54439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-mon[58095]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:08.459 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T07:57:08.368+0000 7f7e03c177c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-27T07:57:08.459 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T07:57:08.402+0000 7f7e03c177c0 -1 osd.6 135 log_to_monitors true 2024-02-27T07:57:08.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[64590]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[64590]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[64590]: from='client.54439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[64590]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[67157]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[67157]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[67157]: from='client.54439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:08 smithi149 ceph-mon[67157]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:57:08.731 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "11/23 daemons upgraded", 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:57:08.732 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: osd.6 (root=default,host=smithi195) is down 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-02-27T07:57:09.119 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 5.18 is active+undersized+degraded, acting [4,1] 2024-02-27T07:57:09.120 INFO:teuthology.orchestra.run.smithi149.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-02-27T07:57:09.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3831470593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:09.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-mon[58095]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-mon[58095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-mon[58095]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:09.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/1884571003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:09.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3831470593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:09.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[67157]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[67157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[67157]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/1884571003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3831470593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[64590]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[64590]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[64590]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:09 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/1884571003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:09.714 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 07:57:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T07:57:09.296+0000 7f7df506f700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:57:10.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: osdmap e138: 8 total, 7 up, 8 in 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[67157]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: osdmap e138: 8 total, 7 up, 8 in 2024-02-27T07:57:10.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:10 smithi149 ceph-mon[64590]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: osdmap e138: 8 total, 7 up, 8 in 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: from='osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:10.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:10 smithi195 ceph-mon[58095]: pgmap v184: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301] boot 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: osdmap e139: 8 total, 8 up, 8 in 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:11.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:11 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301] boot 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: osdmap e139: 8 total, 8 up, 8 in 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:11.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: osd.6 [v2:172.21.15.195:6816/2421276301,v1:172.21.15.195:6817/2421276301] boot 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: osdmap e139: 8 total, 8 up, 8 in 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:11.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:11 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:12.563 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:12 smithi195 ceph-mon[58095]: osdmap e140: 8 total, 8 up, 8 in 2024-02-27T07:57:12.564 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:12 smithi195 ceph-mon[58095]: pgmap v187: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/630 objects degraded (12.063%) 2024-02-27T07:57:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:12 smithi149 ceph-mon[67157]: osdmap e140: 8 total, 8 up, 8 in 2024-02-27T07:57:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:12 smithi149 ceph-mon[67157]: pgmap v187: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/630 objects degraded (12.063%) 2024-02-27T07:57:12.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:12 smithi149 ceph-mon[64590]: osdmap e140: 8 total, 8 up, 8 in 2024-02-27T07:57:12.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:12 smithi149 ceph-mon[64590]: pgmap v187: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/630 objects degraded (12.063%) 2024-02-27T07:57:14.676 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:57:14.203+0000 7fd6feeab700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:14] "GET /metrics HTTP/1.1" 200 38407 "" "Prometheus/2.33.4" 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: pgmap v188: 161 pgs: 13 active+undersized+degraded, 22 active+undersized, 126 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/630 objects degraded (8.413%) 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:15.178 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: pgmap v188: 161 pgs: 13 active+undersized+degraded, 22 active+undersized, 126 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/630 objects degraded (8.413%) 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.179 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: pgmap v188: 161 pgs: 13 active+undersized+degraded, 22 active+undersized, 126 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/630 objects degraded (8.413%) 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:15.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:16.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:16 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:16.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:16 smithi149 ceph-mon[67157]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-27T07:57:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:16 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:16.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:16 smithi149 ceph-mon[64590]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-27T07:57:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:16 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:16 smithi195 ceph-mon[58095]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-27T07:57:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:17 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:17.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:17 smithi149 ceph-mon[67157]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-27T07:57:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:17 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:17.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:17 smithi149 ceph-mon[64590]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-27T07:57:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:17 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:17.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:17 smithi195 ceph-mon[58095]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-27T07:57:18.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:18 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/630 objects degraded (8.413%), 13 pgs degraded) 2024-02-27T07:57:18.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:18 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:57:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:18 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/630 objects degraded (8.413%), 13 pgs degraded) 2024-02-27T07:57:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:18 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:57:18.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:18 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/630 objects degraded (8.413%), 13 pgs degraded) 2024-02-27T07:57:18.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:18 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:57:19.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:19 smithi149 ceph-mon[67157]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:57:19.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:19 smithi149 ceph-mon[64590]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:57:19.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:19 smithi195 ceph-mon[58095]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-27T07:57:21.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:21 smithi149 ceph-mon[64590]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-27T07:57:21.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:21 smithi149 ceph-mon[67157]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-27T07:57:21.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:21 smithi195 ceph-mon[58095]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-02-27T07:57:23.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:23 smithi149 ceph-mon[67157]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-02-27T07:57:23.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:23 smithi149 ceph-mon[64590]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-02-27T07:57:23.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:23 smithi195 ceph-mon[58095]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-02-27T07:57:25.164 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:24] "GET /metrics HTTP/1.1" 200 38407 "" "Prometheus/2.33.4" 2024-02-27T07:57:25.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:25 smithi149 ceph-mon[67157]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:25.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:25 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:25 smithi149 ceph-mon[64590]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:25.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:25 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:25.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:25 smithi195 ceph-mon[58095]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:25.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:25 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:27.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:27 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:27.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:27 smithi149 ceph-mon[67157]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:27 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:27.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:27 smithi149 ceph-mon[64590]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:27 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:27 smithi195 ceph-mon[58095]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:29.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[67157]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:29.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[64590]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:29 smithi195 ceph-mon[58095]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:30.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:30.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:31.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:31.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: Upgrade: osd.7 is safe to restart 2024-02-27T07:57:31.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:31.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: Upgrade: Updating osd.7 2024-02-27T07:57:31.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:31.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:57:31.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:31.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:31 smithi195 ceph-mon[58095]: Deploying daemon osd.7 on smithi195 2024-02-27T07:57:31.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: Upgrade: osd.7 is safe to restart 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: Upgrade: Updating osd.7 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[64590]: Deploying daemon osd.7 on smithi195 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: Upgrade: osd.7 is safe to restart 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:31.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: Upgrade: Updating osd.7 2024-02-27T07:57:31.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:31.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T07:57:31.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:31.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:31 smithi149 ceph-mon[67157]: Deploying daemon osd.7 on smithi195 2024-02-27T07:57:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:32 smithi195 ceph-mon[58095]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:32.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:32 smithi149 ceph-mon[64590]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:32.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:32 smithi149 ceph-mon[67157]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:57:34.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:34 smithi195 ceph-mon[58095]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:34.445 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:34 smithi195 systemd[1]: Stopping Ceph osd.7 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:57:34.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:34 smithi149 ceph-mon[64590]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:34.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:34 smithi149 ceph-mon[67157]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:34.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[37907]: 2024-02-27T07:57:34.444+0000 7f7ebd9a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T07:57:34.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[37907]: 2024-02-27T07:57:34.444+0000 7f7ebd9a8700 -1 osd.7 140 *** Got signal Terminated *** 2024-02-27T07:57:34.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:34 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[37907]: 2024-02-27T07:57:34.444+0000 7f7ebd9a8700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T07:57:35.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:34] "GET /metrics HTTP/1.1" 200 38553 "" "Prometheus/2.33.4" 2024-02-27T07:57:35.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:35 smithi195 ceph-mon[58095]: osd.7 marked itself down and dead 2024-02-27T07:57:35.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:35 smithi149 ceph-mon[67157]: osd.7 marked itself down and dead 2024-02-27T07:57:35.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:35 smithi149 ceph-mon[64590]: osd.7 marked itself down and dead 2024-02-27T07:57:35.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:35 smithi195 podman[67910]: 2024-02-27 07:57:35.671270116 +0000 UTC m=+1.334358225 container died f4759af2e64909645b8aea53570dac5cedad6312ddf8849eda5ae1cf33edf0e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, release=754, distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-27T07:57:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:36 smithi195 ceph-mon[58095]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:36 smithi195 ceph-mon[58095]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-27T07:57:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:36 smithi195 ceph-mon[58095]: osdmap e141: 8 total, 7 up, 8 in 2024-02-27T07:57:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:36 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:36.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:36 smithi195 ceph-mon[58095]: pgmap v200: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:36.465 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:35 smithi195 podman[67910]: 2024-02-27 07:57:35.993613337 +0000 UTC m=+1.656701447 container cleanup f4759af2e64909645b8aea53570dac5cedad6312ddf8849eda5ae1cf33edf0e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, vendor=Red Hat, Inc., io.buildah.version=1.19.8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD) 2024-02-27T07:57:36.465 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:35 smithi195 bash[67910]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7 2024-02-27T07:57:36.465 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[67928]: 2024-02-27 07:57:36.19475234 +0000 UTC m=+0.518287318 container remove f4759af2e64909645b8aea53570dac5cedad6312ddf8849eda5ae1cf33edf0e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, RELEASE=HEAD, ceph=True, io.buildah.version=1.19.8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc.) 2024-02-27T07:57:36.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[67157]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:36.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[67157]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-27T07:57:36.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[67157]: osdmap e141: 8 total, 7 up, 8 in 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[67157]: pgmap v200: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[64590]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[64590]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[64590]: osdmap e141: 8 total, 7 up, 8 in 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:36.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:36 smithi149 ceph-mon[64590]: pgmap v200: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:36.885 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[68016]: 2024-02-27 07:57:36.534676939 +0000 UTC m=+0.018228770 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:36.885 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[68016]: 2024-02-27 07:57:36.662675 +0000 UTC m=+0.146226832 container create a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:57:36.885 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[68016]: 2024-02-27 07:57:36.881595135 +0000 UTC m=+0.365146967 container init a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-27T07:57:36.885 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[68016]: 2024-02-27 07:57:36.885115339 +0000 UTC m=+0.368667182 container start a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:57:37.172 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:36 smithi195 podman[68016]: 2024-02-27 07:57:36.981707855 +0000 UTC m=+0.465259705 container attach a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-02-27T07:57:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:37 smithi195 ceph-mon[58095]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:37.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:37 smithi195 ceph-mon[58095]: osdmap e142: 8 total, 7 up, 8 in 2024-02-27T07:57:37.464 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 podman[68016]: 2024-02-27 07:57:37.172317859 +0000 UTC m=+0.655869693 container died a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T07:57:37.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:37 smithi149 ceph-mon[67157]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:37.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:37 smithi149 ceph-mon[67157]: osdmap e142: 8 total, 7 up, 8 in 2024-02-27T07:57:37.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:37 smithi149 ceph-mon[64590]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-27T07:57:37.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:37 smithi149 ceph-mon[64590]: osdmap e142: 8 total, 7 up, 8 in 2024-02-27T07:57:38.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 podman[68042]: 2024-02-27 07:57:37.768668631 +0000 UTC m=+0.589898746 container remove a19d8c86ee55acada91ea5fc7d0a67330774b2c527a25b0244bb4daf29e331bf (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:57:38.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-27T07:57:38.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 systemd[1]: Stopped Ceph osd.7 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:57:38.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.7.service: Consumed 10.177s CPU time. 2024-02-27T07:57:38.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:37 smithi195 systemd[1]: Starting Ceph osd.7 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T07:57:38.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-mon[58095]: pgmap v202: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:38.482 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.117641344 +0000 UTC m=+0.018392771 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:38.482 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.21959929 +0000 UTC m=+0.120350745 container create 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-27T07:57:38.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:38 smithi149 ceph-mon[64590]: pgmap v202: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:38.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:38 smithi149 ceph-mon[67157]: pgmap v202: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:57:38.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.481674394 +0000 UTC m=+0.382425826 container init 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-27T07:57:38.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.485264111 +0000 UTC m=+0.386015524 container start 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T07:57:38.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.564226528 +0000 UTC m=+0.464977951 container attach 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T07:57:39.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T07:57:39.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T07:57:39.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T07:57:39.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 bash[68146]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate[68161]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-27T07:57:39.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:38 smithi195 podman[68146]: 2024-02-27 07:57:38.946469085 +0000 UTC m=+0.847220500 container died 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-27T07:57:39.465 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:57:39.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:39 smithi195 podman[68278]: 2024-02-27 07:57:39.676932917 +0000 UTC m=+0.725623148 container remove 2425640db7adfa2827951d273662a29b026853f68b461791d74c70a8a3d74e49 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-activate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-27T07:57:39.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:39 smithi195 podman[68344]: 2024-02-27 07:57:39.79939688 +0000 UTC m=+0.018155725 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T07:57:39.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:39 smithi195 podman[68344]: 2024-02-27 07:57:39.911099213 +0000 UTC m=+0.129858043 container create 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-27T07:57:40.112 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (7m) 2m ago 11m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (6m) 30s ago 11m 52.5M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (6m) 2m ago 10m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (5m) 30s ago 16m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (6m) 2m ago 17m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (5m) 2m ago 18m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (4m) 30s ago 16m 38.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (5m) 2m ago 16m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (7m) 2m ago 12m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (6m) 30s ago 12m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (4m) 2m ago 16m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (3m) 2m ago 15m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (2m) 2m ago 15m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (2m) 2m ago 14m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:57:40.113 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (105s) 30s ago 14m 55.2M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (70s) 30s ago 14m 55.4M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (35s) 30s ago 13m 13.6M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a d24e22c91a15 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (13m) 30s ago 13m 61.5M 2992M 17.2.0 e1d6a67b021e f4759af2e649 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (5m) 30s ago 12m 68.8M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (11m) 2m ago 11m 72.6M - 17.2.0 e1d6a67b021e de0c8b9c01d4 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (11m) 30s ago 11m 73.0M - 17.2.0 e1d6a67b021e fa39e7ddda9b 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (10m) 2m ago 10m 74.6M - 17.2.0 e1d6a67b021e 287b2d72b488 2024-02-27T07:57:40.114 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (10m) 30s ago 10m 74.8M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:57:40.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-mon[58095]: from='client.54466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-mon[58095]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-mon[58095]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-mon[58095]: pgmap v203: 161 pgs: 21 active+undersized+degraded, 5 peering, 43 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:40.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-mon[58095]: Health check failed: Degraded data redundancy: 76/630 objects degraded (12.063%), 21 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:40.306 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 podman[68344]: 2024-02-27 07:57:40.205317715 +0000 UTC m=+0.424076557 container init 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-27T07:57:40.306 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 podman[68344]: 2024-02-27 07:57:40.208571005 +0000 UTC m=+0.427329838 container start 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-27T07:57:40.306 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-osd[68363]: -- 172.21.15.195:0/1056719702 <== mon.2 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555f0db41680 con 0x555f0db0b400 2024-02-27T07:57:40.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[64590]: from='client.54466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[64590]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[64590]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[64590]: pgmap v203: 161 pgs: 21 active+undersized+degraded, 5 peering, 43 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:40.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[64590]: Health check failed: Degraded data redundancy: 76/630 objects degraded (12.063%), 21 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[67157]: from='client.54466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[67157]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[67157]: from='client.54478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:40.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[67157]: pgmap v203: 161 pgs: 21 active+undersized+degraded, 5 peering, 43 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2024-02-27T07:57:40.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:40 smithi149 ceph-mon[67157]: Health check failed: Degraded data redundancy: 76/630 objects degraded (12.063%), 21 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 7 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:57:40.531 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:57:40.532 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-27T07:57:40.532 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 12 2024-02-27T07:57:40.532 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:57:40.532 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:57:40.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 bash[68344]: 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee 2024-02-27T07:57:40.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 systemd[1]: Started Ceph osd.7 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T07:57:40.879 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "12/23 daemons upgraded", 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:57:40.880 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:57:41.185 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:40 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T07:57:40.746+0000 7fef9034f7c0 -1 Falling back to public interface 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg peering; Degraded data redundancy: 76/630 objects degraded (12.063%), 21 pgs degraded 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout: osd.7 (root=default,host=smithi195) is down 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.0 is stuck peering for 11m, current state peering, last acting [1,0] 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 76/630 objects degraded (12.063%), 21 pgs degraded 2024-02-27T07:57:41.279 INFO:teuthology.orchestra.run.smithi149.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-02-27T07:57:41.280 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-02-27T07:57:41.281 INFO:teuthology.orchestra.run.smithi149.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-02-27T07:57:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:41 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/1498120156' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:41 smithi195 ceph-mon[58095]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:41.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:41 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/824407768' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:41.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/1498120156' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[67157]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/824407768' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/1498120156' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[64590]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:57:41.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:41 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/824407768' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:57:42.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:42 smithi149 ceph-mon[67157]: pgmap v204: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:42.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:42 smithi149 ceph-mon[64590]: pgmap v204: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:42.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:42 smithi195 ceph-mon[58095]: pgmap v204: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:43.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:43 smithi149 ceph-mon[67157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-27T07:57:43.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:43 smithi149 ceph-mon[64590]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-27T07:57:43.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:43 smithi195 ceph-mon[58095]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-27T07:57:43.693 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:43 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T07:57:43.461+0000 7fef9034f7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-27T07:57:43.693 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:43 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T07:57:43.494+0000 7fef9034f7c0 -1 osd.7 140 log_to_monitors true 2024-02-27T07:57:44.464 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T07:57:44.388+0000 7fef817a7700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T07:57:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-mon[58095]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-mon[58095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:44 smithi195 ceph-mon[58095]: pgmap v205: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:44.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[67157]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[67157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:44.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[67157]: pgmap v205: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[64590]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[64590]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:57:44.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-mon[64590]: pgmap v205: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:45.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:44] "GET /metrics HTTP/1.1" 200 38632 "" "Prometheus/2.33.4" 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: osdmap e143: 8 total, 7 up, 8 in 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: osdmap e143: 8 total, 7 up, 8 in 2024-02-27T07:57:45.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:45.678 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: from='osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: osdmap e143: 8 total, 7 up, 8 in 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:45.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777] boot 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: osdmap e144: 8 total, 8 up, 8 in 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[64590]: pgmap v208: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:46.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777] boot 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: osdmap e144: 8 total, 8 up, 8 in 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:46 smithi149 ceph-mon[67157]: pgmap v208: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: osd.7 [v2:172.21.15.195:6824/309117777,v1:172.21.15.195:6825/309117777] boot 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: osdmap e144: 8 total, 8 up, 8 in 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:46.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:46 smithi195 ceph-mon[58095]: pgmap v208: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-02-27T07:57:47.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:47 smithi195 ceph-mon[58095]: osdmap e145: 8 total, 8 up, 8 in 2024-02-27T07:57:47.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:47 smithi149 ceph-mon[64590]: osdmap e145: 8 total, 8 up, 8 in 2024-02-27T07:57:47.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:47 smithi149 ceph-mon[67157]: osdmap e145: 8 total, 8 up, 8 in 2024-02-27T07:57:48.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:48 smithi149 ceph-mon[64590]: pgmap v210: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-02-27T07:57:48.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:48 smithi149 ceph-mon[67157]: pgmap v210: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-02-27T07:57:48.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:48 smithi195 ceph-mon[58095]: pgmap v210: 161 pgs: 24 active+undersized+degraded, 45 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-02-27T07:57:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: Health check update: Degraded data redundancy: 81/630 objects degraded (12.857%), 24 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:49.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:49 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: Health check update: Degraded data redundancy: 81/630 objects degraded (12.857%), 24 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:49.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: Health check update: Degraded data redundancy: 81/630 objects degraded (12.857%), 24 pgs degraded (PG_DEGRADED) 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:57:49.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:49 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:50.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all osd 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-27T07:57:50.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: Upgrade: Setting require_osd_release to 19 squid 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-27T07:57:50.678 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[67157]: pgmap v211: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/630 objects degraded (0.635%) 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all osd 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-27T07:57:50.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: Upgrade: Setting require_osd_release to 19 squid 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-27T07:57:50.680 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:50 smithi149 ceph-mon[64590]: pgmap v211: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/630 objects degraded (0.635%) 2024-02-27T07:57:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:57:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all osd 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-27T07:57:50.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: Upgrade: Setting require_osd_release to 19 squid 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-27T07:57:50.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:50 smithi195 ceph-mon[58095]: pgmap v211: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 4/630 objects degraded (0.635%) 2024-02-27T07:57:51.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: osdmap e146: 8 total, 8 up, 8 in 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all mds 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: osdmap e146: 8 total, 8 up, 8 in 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all mds 2024-02-27T07:57:51.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:51 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: osdmap e146: 8 total, 8 up, 8 in 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all mds 2024-02-27T07:57:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:51 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:53.307 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: Upgrade: Updating rgw.foo.smithi149.akhivq (1/4) 2024-02-27T07:57:53.307 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:53.307 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:53.307 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:53.307 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[67157]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: Upgrade: Updating rgw.foo.smithi149.akhivq (1/4) 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:57:53.308 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:53 smithi149 ceph-mon[64590]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-27T07:57:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: Upgrade: Updating rgw.foo.smithi149.akhivq (1/4) 2024-02-27T07:57:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.akhivq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:53.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:53.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: Deploying daemon rgw.foo.smithi149.akhivq on smithi149 2024-02-27T07:57:53.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:53 smithi195 ceph-mon[58095]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-27T07:57:54.137 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:54 smithi149 ceph-mon[67157]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/630 objects degraded (0.635%), 2 pgs degraded) 2024-02-27T07:57:54.137 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:54 smithi149 ceph-mon[67157]: Cluster is now healthy 2024-02-27T07:57:54.137 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:54 smithi149 ceph-mon[64590]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/630 objects degraded (0.635%), 2 pgs degraded) 2024-02-27T07:57:54.137 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:54 smithi149 ceph-mon[64590]: Cluster is now healthy 2024-02-27T07:57:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:54 smithi195 ceph-mon[58095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/630 objects degraded (0.635%), 2 pgs degraded) 2024-02-27T07:57:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:54 smithi195 ceph-mon[58095]: Cluster is now healthy 2024-02-27T07:57:55.029 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:57:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:57:54] "GET /metrics HTTP/1.1" 200 38632 "" "Prometheus/2.33.4" 2024-02-27T07:57:55.353 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:55 smithi149 ceph-mon[67157]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:55.353 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:55 smithi149 ceph-mon[64590]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:55 smithi195 ceph-mon[58095]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[64590]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:56 smithi149 ceph-mon[67157]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:57:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:56 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:57:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:56 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:56 smithi195 ceph-mon[58095]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:57:58.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: Upgrade: Updating rgw.smpl.smithi149.bmexvh (2/4) 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[67157]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: Upgrade: Updating rgw.smpl.smithi149.bmexvh (2/4) 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:58.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:58.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:57:58.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:57:58 smithi149 ceph-mon[64590]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:57:58.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: Upgrade: Updating rgw.smpl.smithi149.bmexvh (2/4) 2024-02-27T07:57:58.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:57:58.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.bmexvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:57:58.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:57:58.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: Deploying daemon rgw.smpl.smithi149.bmexvh on smithi149 2024-02-27T07:57:58.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:57:58 smithi195 ceph-mon[58095]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:58:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:00.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:00.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:00.428 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:00 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:00 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:01.270 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:01 smithi149 ceph-mon[64590]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-02-27T07:58:01.270 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:01 smithi149 ceph-mon[67157]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-02-27T07:58:01.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:01 smithi195 ceph-mon[58095]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-02-27T07:58:02.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[64590]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 87 B/s wr, 86 op/s 2024-02-27T07:58:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:02 smithi149 ceph-mon[67157]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 87 B/s wr, 86 op/s 2024-02-27T07:58:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:02 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:02.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:02 smithi195 ceph-mon[58095]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 87 B/s wr, 86 op/s 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: Upgrade: Updating rgw.foo.smithi195.xuyrge (3/4) 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:58:04.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:04 smithi195 ceph-mon[58095]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-02-27T07:58:04.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: Upgrade: Updating rgw.foo.smithi195.xuyrge (3/4) 2024-02-27T07:58:04.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:04.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:04.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[64590]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: Upgrade: Updating rgw.foo.smithi195.xuyrge (3/4) 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi195.xuyrge", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: Deploying daemon rgw.foo.smithi195.xuyrge on smithi195 2024-02-27T07:58:04.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-mon[67157]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-02-27T07:58:05.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:04] "GET /metrics HTTP/1.1" 200 38636 "" "Prometheus/2.33.4" 2024-02-27T07:58:06.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:06 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:06.400 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:06 smithi195 ceph-mon[58095]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:06.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:06 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:06.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:06 smithi149 ceph-mon[67157]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:06 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:06.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:06 smithi149 ceph-mon[64590]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:08.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[67157]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:08 smithi149 ceph-mon[64590]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:08.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:08 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:08 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:08.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:08 smithi195 ceph-mon[58095]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:10.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:10 smithi195 ceph-mon[58095]: Upgrade: Updating rgw.smpl.smithi195.lvkuxs (4/4) 2024-02-27T07:58:10.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:10.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:10.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:10 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:10.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:10 smithi195 ceph-mon[58095]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:58:10.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[67157]: Upgrade: Updating rgw.smpl.smithi195.lvkuxs (4/4) 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[67157]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[64590]: Upgrade: Updating rgw.smpl.smithi195.lvkuxs (4/4) 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi195.lvkuxs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:10 smithi149 ceph-mon[64590]: Deploying daemon rgw.smpl.smithi195.lvkuxs on smithi195 2024-02-27T07:58:11.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:11 smithi195 ceph-mon[58095]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 170 B/s wr, 239 op/s 2024-02-27T07:58:11.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:11 smithi149 ceph-mon[67157]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 170 B/s wr, 239 op/s 2024-02-27T07:58:11.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:11 smithi149 ceph-mon[64590]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 155 KiB/s rd, 170 B/s wr, 239 op/s 2024-02-27T07:58:11.635 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:58:12.281 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (7m) 2m ago 12m 24.9M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (7m) 27s ago 12m 53.0M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 running (6m) 2m ago 11m 52.7M - 3.5 e1d6a67b021e 1578472612fa 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (6m) 27s ago 16m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (6m) 2m ago 18m 506M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (6m) 2m ago 18m 49.8M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (5m) 27s ago 17m 41.2M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (5m) 2m ago 17m 40.4M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (7m) 2m ago 13m 17.5M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (7m) 27s ago 13m 17.4M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (4m) 2m ago 16m 44.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (4m) 2m ago 16m 53.1M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (3m) 2m ago 15m 41.0M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (2m) 2m ago 15m 13.6M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (2m) 27s ago 15m 56.1M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:58:12.282 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (102s) 27s ago 14m 57.2M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (67s) 27s ago 14m 42.5M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a d24e22c91a15 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (32s) 27s ago 13m 12.7M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 21f14d203379 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (6m) 27s ago 13m 68.9M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 starting - - - - 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 starting - - - - 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 starting - - - - 2024-02-27T07:58:12.283 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (11m) 27s ago 11m 75.0M - 17.2.0 e1d6a67b021e 1f460185098e 2024-02-27T07:58:12.375 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:12 smithi195 ceph-mon[58095]: from='client.44619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.375 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:12 smithi195 ceph-mon[58095]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.375 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:12 smithi195 ceph-mon[58095]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 255 B/s wr, 181 op/s 2024-02-27T07:58:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[67157]: from='client.44619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[67157]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[67157]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 255 B/s wr, 181 op/s 2024-02-27T07:58:12.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[64590]: from='client.44619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[64590]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:12.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:12 smithi149 ceph-mon[64590]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 255 B/s wr, 181 op/s 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:58:12.682 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 16 2024-02-27T07:58:12.683 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:58:12.683 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T07:58:13.018 INFO:teuthology.orchestra.run.smithi149.stdout: "osd", 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "13/23 daemons upgraded", 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Currently upgrading rgw daemons", 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": false 2024-02-27T07:58:13.019 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:58:13.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[64590]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3176467680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:13.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[64590]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[67157]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3176467680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[67157]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:13 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:13 smithi195 ceph-mon[58095]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:13 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3176467680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:13 smithi195 ceph-mon[58095]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:13 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:13.480 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_OK 2024-02-27T07:58:14.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/710012242' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:58:14.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:14.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[67157]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-27T07:58:14.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/710012242' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:58:14.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:14.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-mon[64590]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-27T07:58:14.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:14 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/710012242' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T07:58:14.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:14 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:14.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:14 smithi195 ceph-mon[58095]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-02-27T07:58:15.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:14] "GET /metrics HTTP/1.1" 200 38632 "" "Prometheus/2.33.4" 2024-02-27T07:58:16.566 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:16 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:16.566 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:16 smithi195 ceph-mon[58095]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-27T07:58:16.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:16 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:16.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:16 smithi149 ceph-mon[67157]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-27T07:58:16.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:16 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:16.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:16 smithi149 ceph-mon[64590]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 255 B/s wr, 225 op/s 2024-02-27T07:58:18.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.176 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:17 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:18.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:17 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[67157]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:18 smithi149 ceph-mon[64590]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:18 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:19.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:18 smithi195 ceph-mon[58095]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:20 smithi195 ceph-mon[58095]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:20.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:20 smithi149 ceph-mon[64590]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:20.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:20 smithi149 ceph-mon[67157]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-27T07:58:22.546 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: Detected new or changed devices on smithi149 2024-02-27T07:58:22.546 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.546 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.546 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: Detected new or changed devices on smithi195 2024-02-27T07:58:22.546 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all rgw 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.547 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]: dispatch 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]': finished 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]: dispatch 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]': finished 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]: dispatch 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]': finished 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]: dispatch 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]': finished 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.548 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: Detected new or changed devices on smithi149 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: Detected new or changed devices on smithi195 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.549 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all rgw 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]': finished 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]': finished 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]': finished 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]': finished 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all rbd-mirror 2024-02-27T07:58:22.550 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: Upgrade: Setting container_image for all ceph-exporter 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[64590]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 97 op/s 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all rbd-mirror 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: Upgrade: Setting container_image for all ceph-exporter 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.551 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:22 smithi149 ceph-mon[67157]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 97 op/s 2024-02-27T07:58:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: Detected new or changed devices on smithi149 2024-02-27T07:58:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: Detected new or changed devices on smithi195 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all rgw 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.akhivq"}]': finished 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi195.xuyrge"}]': finished 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.bmexvh"}]': finished 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi195.lvkuxs"}]': finished 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all rbd-mirror 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: Upgrade: Setting container_image for all ceph-exporter 2024-02-27T07:58:22.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:22.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:22 smithi195 ceph-mon[58095]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 170 B/s wr, 97 op/s 2024-02-27T07:58:24.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: Upgrade: Updating iscsi.foo.smithi149.gjsccy 2024-02-27T07:58:24.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:24.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[67157]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: Upgrade: Updating iscsi.foo.smithi149.gjsccy 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:58:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-mon[64590]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: Upgrade: Updating iscsi.foo.smithi149.gjsccy 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi149.gjsccy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: Deploying daemon iscsi.foo.smithi149.gjsccy on smithi149 2024-02-27T07:58:24.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:24 smithi195 ceph-mon[58095]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-02-27T07:58:25.118 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:24] "GET /metrics HTTP/1.1" 200 38632 "" "Prometheus/2.33.4" 2024-02-27T07:58:26.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:26 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:26.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:26 smithi149 ceph-mon[67157]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 57 op/s 2024-02-27T07:58:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:26 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:26.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:26 smithi149 ceph-mon[64590]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 57 op/s 2024-02-27T07:58:26.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:26 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:26.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:26 smithi195 ceph-mon[58095]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 57 op/s 2024-02-27T07:58:28.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:28 smithi149 ceph-mon[67157]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:28.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:28 smithi149 ceph-mon[64590]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:28.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:28 smithi195 ceph-mon[58095]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:29.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:29 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:29.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:29 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:29.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:29 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:30.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:30 smithi149 ceph-mon[67157]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:30.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:30 smithi149 ceph-mon[64590]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:30.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:30 smithi195 ceph-mon[58095]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:32.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:32 smithi149 ceph-mon[67157]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:32.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:32 smithi149 ceph-mon[64590]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:32.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:32 smithi195 ceph-mon[58095]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:34.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:34 smithi149 ceph-mon[67157]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:34.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:34 smithi149 ceph-mon[64590]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:34.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:34 smithi195 ceph-mon[58095]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:35.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:34] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.33.4" 2024-02-27T07:58:36.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:36 smithi149 ceph-mon[64590]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:36.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:36 smithi149 ceph-mon[64590]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:36.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:36 smithi149 ceph-mon[67157]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:36.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:36 smithi149 ceph-mon[67157]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:36 smithi195 ceph-mon[58095]: from='client.35138 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:36.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:36 smithi195 ceph-mon[58095]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:58:37.152 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T07:58:36.790+0000 7fd6cf643700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T07:58:37.863 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:37 smithi195 ceph-mon[58095]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T07:58:37.863 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:37 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:37.863 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:37 smithi195 ceph-mon[58095]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:38.127 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[67157]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T07:58:38.127 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:38.127 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[67157]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:38.127 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[64590]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T07:58:38.128 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:38.128 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:37 smithi149 ceph-mon[64590]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:38.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:38 smithi149 ceph-mon[67157]: Health check failed: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-27T07:58:38.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:38 smithi149 ceph-mon[64590]: Health check failed: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-27T07:58:39.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:38 smithi195 ceph-mon[58095]: Health check failed: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-27T07:58:39.916 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:39 smithi149 ceph-mon[67157]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:39.917 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:39 smithi149 ceph-mon[64590]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:40.112 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:39 smithi195 ceph-mon[58095]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: Checking dashboard <-> RGW credentials 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: Checking dashboard <-> RGW credentials 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.928 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-27T07:58:41.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:41.929 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:41 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: Checking dashboard <-> RGW credentials 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:41.965 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:41 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:42.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:42 smithi149 ceph-mon[67157]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:58:42.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:42 smithi149 ceph-mon[64590]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:58:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:42 smithi195 ceph-mon[58095]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T07:58:43.848 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T07:58:43.849 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:43 smithi149 ceph-mon[64590]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:43.849 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:43 smithi149 ceph-mon[67157]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:43.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:43 smithi195 ceph-mon[58095]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 0 op/s 2024-02-27T07:58:44.207 INFO:teuthology.orchestra.run.smithi149.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed." 2024-02-27T07:58:44.877 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch ps' 2024-02-27T07:58:45.150 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:44] "GET /metrics HTTP/1.1" 200 38783 "" "Prometheus/2.33.4" 2024-02-27T07:58:45.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[67157]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:45.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[64590]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:45.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:45 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:45 smithi195 ceph-mon[58095]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:58:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:45 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:46.314 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:46 smithi149 ceph-mon[67157]: from='client.54670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:46.314 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:46 smithi149 ceph-mon[67157]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:58:46.315 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:46 smithi149 ceph-mon[64590]: from='client.54670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:46.315 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:46 smithi149 ceph-mon[64590]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:58:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:46 smithi195 ceph-mon[58095]: from='client.54670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:46.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:46 smithi195 ceph-mon[58095]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (8m) 5s ago 12m 24.2M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (7m) 6s ago 12m 53.0M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 unknown 5s ago 11m - - 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (6m) 6s ago 17m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:58:46.720 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (7m) 5s ago 19m 513M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (6m) 5s ago 19m 60.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (6m) 6s ago 17m 41.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (6m) 5s ago 17m 47.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (8m) 5s ago 13m 17.1M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (7m) 6s ago 13m 17.3M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (5m) 5s ago 17m 54.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (4m) 5s ago 16m 66.4M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (4m) 5s ago 16m 49.5M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (3m) 5s ago 16m 53.9M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (2m) 6s ago 15m 63.0M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (2m) 6s ago 15m 60.7M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (101s) 6s ago 14m 46.6M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a d24e22c91a15 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (66s) 6s ago 14m 48.5M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 21f14d203379 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (7m) 6s ago 13m 70.2M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:58:46.721 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (51s) 5s ago 12m 75.5M - 19.0.0-1736-g635ee685 c4eb0759a73a 0ef7c9eb947a 2024-02-27T07:58:46.722 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (39s) 6s ago 12m 77.6M - 19.0.0-1736-g635ee685 c4eb0759a73a 727fe07a0712 2024-02-27T07:58:46.722 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (45s) 5s ago 12m 76.8M - 19.0.0-1736-g635ee685 c4eb0759a73a 5ef88f1d281f 2024-02-27T07:58:46.722 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (34s) 6s ago 12m 75.3M - 19.0.0-1736-g635ee685 c4eb0759a73a 2ea4f17435ef 2024-02-27T07:58:47.332 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions' 2024-02-27T07:58:48.350 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:48 smithi149 ceph-mon[67157]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:48.350 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:48 smithi149 ceph-mon[67157]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:48.350 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:48 smithi149 ceph-mon[64590]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:48.350 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:48 smithi149 ceph-mon[64590]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:48.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:48 smithi195 ceph-mon[58095]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:58:48.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:48 smithi195 ceph-mon[58095]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:49.373 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3839622032' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:58:49.373 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/165244825' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.374 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[64590]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.374 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3839622032' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:58:49.374 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/165244825' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.374 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:49 smithi149 ceph-mon[67157]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:49 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3839622032' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-27T07:58:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:49 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/165244825' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:49 smithi195 ceph-mon[58095]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]: dispatch 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T07:58:49.768 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 17 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T07:58:49.769 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T07:58:50.377 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[67157]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[67157]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]': finished 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[67157]: osdmap e147: 8 total, 8 up, 8 in 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]: dispatch 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/732930728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[64590]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[64590]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]': finished 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[64590]: osdmap e147: 8 total, 8 up, 8 in 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]: dispatch 2024-02-27T07:58:50.378 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:50 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/732930728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:50.403 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-27T07:58:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:50 smithi195 ceph-mon[58095]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-02-27T07:58:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:50 smithi195 ceph-mon[58095]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/2510442489"}]': finished 2024-02-27T07:58:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:50 smithi195 ceph-mon[58095]: osdmap e147: 8 total, 8 up, 8 in 2024-02-27T07:58:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:50 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]: dispatch 2024-02-27T07:58:50.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:50 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/732930728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T07:58:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]': finished 2024-02-27T07:58:51.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[67157]: osdmap e148: 8 total, 8 up, 8 in 2024-02-27T07:58:51.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]: dispatch 2024-02-27T07:58:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]': finished 2024-02-27T07:58:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[64590]: osdmap e148: 8 total, 8 up, 8 in 2024-02-27T07:58:51.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:51 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]: dispatch 2024-02-27T07:58:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:51 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/871037898' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6801/806323170"}]': finished 2024-02-27T07:58:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:51 smithi195 ceph-mon[58095]: osdmap e148: 8 total, 8 up, 8 in 2024-02-27T07:58:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:51 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]: dispatch 2024-02-27T07:58:52.137 INFO:teuthology.orchestra.run.smithi149.stdout:wait for servicemap items w/ changing names to refresh 2024-02-27T07:58:52.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[67157]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-27T07:58:52.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]': finished 2024-02-27T07:58:52.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[67157]: osdmap e149: 8 total, 8 up, 8 in 2024-02-27T07:58:52.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]: dispatch 2024-02-27T07:58:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[64590]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-27T07:58:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]': finished 2024-02-27T07:58:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[64590]: osdmap e149: 8 total, 8 up, 8 in 2024-02-27T07:58:52.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:52 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]: dispatch 2024-02-27T07:58:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:52 smithi195 ceph-mon[58095]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-02-27T07:58:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:52 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3039091426' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/85508457"}]': finished 2024-02-27T07:58:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:52 smithi195 ceph-mon[58095]: osdmap e149: 8 total, 8 up, 8 in 2024-02-27T07:58:52.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:52 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]: dispatch 2024-02-27T07:58:52.924 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'sleep 60' 2024-02-27T07:58:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:53 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]': finished 2024-02-27T07:58:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:53 smithi195 ceph-mon[58095]: osdmap e150: 8 total, 8 up, 8 in 2024-02-27T07:58:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:53 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/857235492' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:53 smithi195 ceph-mon[58095]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]': finished 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[67157]: osdmap e150: 8 total, 8 up, 8 in 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/857235492' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[67157]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/1870274374' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:6800/806323170"}]': finished 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[64590]: osdmap e150: 8 total, 8 up, 8 in 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/857235492' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:53.520 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:53 smithi149 ceph-mon[64590]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]: dispatch 2024-02-27T07:58:54.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[67157]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:58:54.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[67157]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]': finished 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[67157]: osdmap e151: 8 total, 8 up, 8 in 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]: dispatch 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[64590]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[64590]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]': finished 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[64590]: osdmap e151: 8 total, 8 up, 8 in 2024-02-27T07:58:54.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]: dispatch 2024-02-27T07:58:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:54 smithi195 ceph-mon[58095]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-02-27T07:58:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:54 smithi195 ceph-mon[58095]: from='client.? ' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1477408981"}]': finished 2024-02-27T07:58:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:54 smithi195 ceph-mon[58095]: osdmap e151: 8 total, 8 up, 8 in 2024-02-27T07:58:54.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:54 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]: dispatch 2024-02-27T07:58:55.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:58:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:58:54] "GET /metrics HTTP/1.1" 200 38783 "" "Prometheus/2.33.4" 2024-02-27T07:58:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:55 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]': finished 2024-02-27T07:58:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:55 smithi195 ceph-mon[58095]: osdmap e152: 8 total, 8 up, 8 in 2024-02-27T07:58:55.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:55 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]': finished 2024-02-27T07:58:55.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:55 smithi149 ceph-mon[67157]: osdmap e152: 8 total, 8 up, 8 in 2024-02-27T07:58:55.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:55 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2858994675' entity='client.iscsi.foo.smithi149.gjsccy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.149:0/1836389262"}]': finished 2024-02-27T07:58:55.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:55 smithi149 ceph-mon[64590]: osdmap e152: 8 total, 8 up, 8 in 2024-02-27T07:58:56.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:56 smithi149 ceph-mon[64590]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:56.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:56 smithi149 ceph-mon[67157]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:56.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:56 smithi195 ceph-mon[58095]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:58:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:57 smithi195 ceph-mon[58095]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-02-27T07:58:57.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:57 smithi149 ceph-mon[67157]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-02-27T07:58:57.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:57 smithi149 ceph-mon[64590]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-02-27T07:58:58.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:58 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:58.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:58 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:58.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:58 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:58:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:59 smithi195 ceph-mon[58095]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:58:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:58:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:59.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:59 smithi149 ceph-mon[67157]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:58:59.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:58:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:58:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:59 smithi149 ceph-mon[64590]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-27T07:58:59.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:58:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:02.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:02 smithi149 ceph-mon[67157]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:02.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:02 smithi149 ceph-mon[64590]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:02.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:02 smithi195 ceph-mon[58095]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:04.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:04 smithi149 ceph-mon[64590]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:59:04.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:04 smithi149 ceph-mon[67157]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:59:04.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:04 smithi195 ceph-mon[58095]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-27T07:59:05.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:04] "GET /metrics HTTP/1.1" 200 38787 "" "Prometheus/2.33.4" 2024-02-27T07:59:06.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:06 smithi149 ceph-mon[64590]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-02-27T07:59:06.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:06 smithi149 ceph-mon[67157]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-02-27T07:59:06.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:06 smithi195 ceph-mon[58095]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-02-27T07:59:08.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:08 smithi149 ceph-mon[67157]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:08.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:08 smithi149 ceph-mon[64590]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:08.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:08 smithi195 ceph-mon[58095]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:09.412 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:09 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:09.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:09 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:09.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:09 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:10.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:10 smithi149 ceph-mon[67157]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:10.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:10 smithi149 ceph-mon[64590]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:10.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:10 smithi195 ceph-mon[58095]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:12.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:12 smithi149 ceph-mon[64590]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:12.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:12 smithi149 ceph-mon[67157]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:12.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:12 smithi195 ceph-mon[58095]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:14.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:14 smithi149 ceph-mon[67157]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:14.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:14 smithi149 ceph-mon[64590]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:14.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:14 smithi195 ceph-mon[58095]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:15.112 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:14] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.33.4" 2024-02-27T07:59:15.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:15 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:15.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:15 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:15.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:15 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:16.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:16 smithi149 ceph-mon[67157]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:16.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:16 smithi149 ceph-mon[64590]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:16.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:16 smithi195 ceph-mon[58095]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:18.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:18 smithi149 ceph-mon[67157]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:18.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:18 smithi149 ceph-mon[64590]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:18.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:18 smithi195 ceph-mon[58095]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:19 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:19.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:19 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:19.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:19 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:20.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:20 smithi149 ceph-mon[67157]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:20.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:20 smithi149 ceph-mon[64590]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:20.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:20 smithi195 ceph-mon[58095]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:22.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:22 smithi149 ceph-mon[67157]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:22.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:22 smithi149 ceph-mon[64590]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:22.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:22 smithi195 ceph-mon[58095]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:24.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:24 smithi149 ceph-mon[67157]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-27T07:59:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:24 smithi149 ceph-mon[64590]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-27T07:59:24.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:24 smithi195 ceph-mon[58095]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-27T07:59:25.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:24] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.33.4" 2024-02-27T07:59:26.424 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:26 smithi149 ceph-mon[67157]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:26.424 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:26 smithi149 ceph-mon[64590]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:26.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:26 smithi195 ceph-mon[58095]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:28.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:28 smithi149 ceph-mon[64590]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:28.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:28 smithi149 ceph-mon[67157]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:28.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:28 smithi195 ceph-mon[58095]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:29.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:29 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:29.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:29 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:29 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:30.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:30 smithi149 ceph-mon[64590]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:30.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:30 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:30.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:30 smithi149 ceph-mon[67157]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:30.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:30 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:30 smithi195 ceph-mon[58095]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:30.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:30 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:32.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:32 smithi149 ceph-mon[64590]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:32.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:32 smithi149 ceph-mon[67157]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:32.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:32 smithi195 ceph-mon[58095]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:34.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:34 smithi149 ceph-mon[67157]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:34.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:34 smithi149 ceph-mon[64590]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:34.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:34 smithi195 ceph-mon[58095]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:35.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:34] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.33.4" 2024-02-27T07:59:36.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:36 smithi149 ceph-mon[64590]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:36.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:36 smithi149 ceph-mon[67157]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:36.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:36 smithi195 ceph-mon[58095]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:38.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:38 smithi149 ceph-mon[64590]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:38.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:38 smithi149 ceph-mon[67157]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:38.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:38 smithi195 ceph-mon[58095]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:39.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:39 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:39.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:39 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:39.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:39 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:40.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:40 smithi149 ceph-mon[64590]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:40.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:40 smithi149 ceph-mon[67157]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:40.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:40 smithi195 ceph-mon[58095]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:42.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[67157]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:42.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:59:42.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:59:42.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:59:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[64590]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:59:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:59:42.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:42 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:59:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:42 smithi195 ceph-mon[58095]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:42 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T07:59:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:42 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T07:59:42.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:42 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' 2024-02-27T07:59:43.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:43 smithi149 ceph-mon[67157]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-27T07:59:43.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:43 smithi149 ceph-mon[67157]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:43.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:43 smithi149 ceph-mon[64590]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-27T07:59:43.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:43 smithi149 ceph-mon[64590]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:43.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:43 smithi195 ceph-mon[58095]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-27T07:59:43.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:43 smithi195 ceph-mon[58095]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:44.426 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:44 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:44.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:44 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:44.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:44 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:45.176 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:44] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.33.4" 2024-02-27T07:59:45.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:45 smithi195 ceph-mon[58095]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:45.679 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:45 smithi149 ceph-mon[64590]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:45.679 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:45 smithi149 ceph-mon[67157]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:47.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:47 smithi195 ceph-mon[58095]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:47.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:47 smithi149 ceph-mon[67157]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:47.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:47 smithi149 ceph-mon[64590]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:49 smithi195 ceph-mon[58095]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:49.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:49 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:49.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:49 smithi149 ceph-mon[67157]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:49.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:49 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:49 smithi149 ceph-mon[64590]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-02-27T07:59:49.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:49 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:51.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:51 smithi195 ceph-mon[58095]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:51.676 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:51 smithi149 ceph-mon[64590]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:51.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:51 smithi149 ceph-mon[67157]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:53 smithi195 ceph-mon[58095]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:59:53.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:53 smithi149 ceph-mon[67157]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:59:53.677 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:53 smithi149 ceph-mon[64590]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-27T07:59:54.918 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 07:59:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:07:59:54] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.33.4" 2024-02-27T07:59:55.313 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch ps' 2024-02-27T07:59:55.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:55 smithi195 ceph-mon[58095]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:55.611 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:55 smithi149 ceph-mon[64590]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:55.612 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:55 smithi149 ceph-mon[67157]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.a smithi149 *:9093,9094 running (9m) 76s ago 14m 24.2M - 0.23.0 ba2b418f427c 8e272eb9189d 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.a smithi195 *:3000 running (8m) 76s ago 13m 53.0M - 8.3.5 dad864ee21e9 9914d359e302 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:iscsi.foo.smithi149.gjsccy smithi149 unknown 76s ago 13m - - 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.x smithi195 *:8443,9283,8765 running (8m) 76s ago 18m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 5408d94e7c8a 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.y smithi149 *:8443,9283,8765 running (8m) 76s ago 20m 513M - 19.0.0-1736-g635ee685 c4eb0759a73a c790b627593b 2024-02-27T07:59:57.241 INFO:teuthology.orchestra.run.smithi149.stdout:mon.a smithi149 running (7m) 76s ago 20m 60.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 1865cab475cf 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:mon.b smithi195 running (7m) 76s ago 18m 41.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 2163ea012c4e 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:mon.c smithi149 running (7m) 76s ago 19m 47.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a b393c5235d2f 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.a smithi149 *:9100 running (9m) 76s ago 14m 17.1M - 1.3.1 1dbe0e931976 73faee7fa866 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.b smithi195 *:9100 running (9m) 76s ago 14m 17.3M - 1.3.1 1dbe0e931976 54ca3ae0e6d5 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (6m) 76s ago 18m 54.8M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 6d66c6d3e8b5 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 running (5m) 76s ago 17m 66.4M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 989ce187fad2 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (5m) 76s ago 17m 49.5M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a 5f7f45342b59 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (4m) 76s ago 17m 53.9M 4096M 19.0.0-1736-g635ee685 c4eb0759a73a edb3d7993300 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi195 running (4m) 76s ago 16m 63.0M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 482e1c8ae05f 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi195 running (3m) 76s ago 16m 60.7M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 258882dddff0 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi195 running (2m) 76s ago 16m 46.6M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a d24e22c91a15 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi195 running (2m) 76s ago 15m 48.5M 2992M 19.0.0-1736-g635ee685 c4eb0759a73a 21f14d203379 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.a smithi195 *:9095 running (8m) 76s ago 14m 70.2M - 2.33.4 514e6a882f6e 170b78cb18ef 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi149.akhivq smithi149 *:8000 running (2m) 76s ago 13m 75.5M - 19.0.0-1736-g635ee685 c4eb0759a73a 0ef7c9eb947a 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.foo.smithi195.xuyrge smithi195 *:8000 running (110s) 76s ago 13m 77.6M - 19.0.0-1736-g635ee685 c4eb0759a73a 727fe07a0712 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi149.bmexvh smithi149 *:80 running (116s) 76s ago 13m 76.8M - 19.0.0-1736-g635ee685 c4eb0759a73a 5ef88f1d281f 2024-02-27T07:59:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:rgw.smpl.smithi195.lvkuxs smithi195 *:80 running (104s) 76s ago 13m 75.3M - 19.0.0-1736-g635ee685 c4eb0759a73a 2ea4f17435ef 2024-02-27T07:59:57.427 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:57 smithi149 ceph-mon[67157]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:57.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:57 smithi149 ceph-mon[64590]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:57.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:57 smithi195 ceph-mon[58095]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T07:59:57.981 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions' 2024-02-27T07:59:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:59 smithi195 ceph-mon[58095]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:59:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:59 smithi195 ceph-mon[58095]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:59 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:59.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 07:59:59 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:59.560 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[67157]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:59:59.560 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[67157]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[64590]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[64590]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T07:59:59.561 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 07:59:59 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: "mon": { 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr": { 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T08:00:00.031 INFO:teuthology.orchestra.run.smithi149.stdout: "osd": { 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw": { 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: "overall": { 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 17 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-02-27T08:00:00.032 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T08:00:00.363 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:00.363 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.363 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T08:00:00.363 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T08:00:00.363 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Upgrade daemon: iscsi.foo.smithi149.gjsccy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Creating ceph-iscsi config... 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d448a3a16\x2dd543\x2d11ee\x2d95c3\x2d87774f69a715.slice/ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Non-zero exit code 1 from systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: Traceback (most recent call last): 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: return _run_code(code, main_globals, None, 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T08:00:00.364 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: exec(code, run_globals) 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1034, in deploy_daemon 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1170, in deploy_daemon_units 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: RuntimeError: Failed command: systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.365 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/3510316915' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Upgrade daemon: iscsi.foo.smithi149.gjsccy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Creating ceph-iscsi config... 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d448a3a16\x2dd543\x2d11ee\x2d95c3\x2d87774f69a715.slice/ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Non-zero exit code 1 from systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T08:00:00.366 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: Traceback (most recent call last): 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: return _run_code(code, main_globals, None, 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: exec(code, run_globals) 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1034, in deploy_daemon 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1170, in deploy_daemon_units 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T08:00:00.367 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: RuntimeError: Failed command: systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.368 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.368 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:00 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/3510316915' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:00.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Upgrade daemon: iscsi.foo.smithi149.gjsccy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Creating ceph-iscsi config... 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d448a3a16\x2dd543\x2d11ee\x2d95c3\x2d87774f69a715.slice/ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Non-zero exit code 1 from systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: Traceback (most recent call last): 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T08:00:00.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: return _run_code(code, main_globals, None, 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: exec(code, run_globals) 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1034, in deploy_daemon 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1170, in deploy_daemon_units 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: RuntimeError: Failed command: systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:00.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:00 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/3510316915' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:00.679 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch upgrade status' 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "in_progress": true, 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "services_complete": [ 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "osd", 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "mgr", 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "rgw", 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "mon" 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-02-27T08:00:02.685 INFO:teuthology.orchestra.run.smithi149.stdout: "progress": "17/23 daemons upgraded", 2024-02-27T08:00:02.686 INFO:teuthology.orchestra.run.smithi149.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed.", 2024-02-27T08:00:02.686 INFO:teuthology.orchestra.run.smithi149.stdout: "is_paused": true 2024-02-27T08:00:02.686 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T08:00:03.018 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:02 smithi149 ceph-mon[64590]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:03.019 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:02 smithi149 ceph-mon[67157]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:03.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:02 smithi195 ceph-mon[58095]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:03.335 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph health detail' 2024-02-27T08:00:03.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:03 smithi149 ceph-mon[67157]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:03.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:03 smithi149 ceph-mon[64590]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:04.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:03 smithi195 ceph-mon[58095]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:04.857 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:04 smithi149 ceph-mon[64590]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:04.857 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:04 smithi149 ceph-mon[67157]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:04.858 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:08:00:04] "GET /metrics HTTP/1.1" 200 38784 "" "Prometheus/2.33.4" 2024-02-27T08:00:05.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:04 smithi195 ceph-mon[58095]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout: daemon iscsi.foo.smithi149.gjsccy on smithi149 is in unknown state 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi149.gjsccy on host smithi149 failed. 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrade daemon: iscsi.foo.smithi149.gjsccy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi149.gjsccy ... 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout:Creating ceph-iscsi config... 2024-02-27T08:00:05.348 INFO:teuthology.orchestra.run.smithi149.stdout:Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/iscsi-gateway.cfg 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:Write file: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/iscsi.foo.smithi149.gjsccy/tcmu-runner-entrypoint.sh 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d448a3a16\x2dd543\x2d11ee\x2d95c3\x2d87774f69a715.slice/ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:Non-zero exit code 1 from systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:systemctl: stderr Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:systemctl: stderr See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout:Traceback (most recent call last): 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: return _run_code(code, main_globals, None, 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: exec(code, run_globals) 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5527, in 2024-02-27T08:00:05.349 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 5515, in main 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3043, in command_deploy_from 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3078, in _common_deploy 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 3098, in _deploy_daemon_container 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1034, in deploy_daemon 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/__main__.py", line 1170, in deploy_daemon_units 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout: File "/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/cephadm.5ac861bc1362dfdd3d29f12c007ea27e880cbe64f656cdd1beb235731353446b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout:RuntimeError: Failed command: systemctl start ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy: Job for ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service failed because the control process exited with error code. 2024-02-27T08:00:05.350 INFO:teuthology.orchestra.run.smithi149.stdout:See "systemctl status ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" and "journalctl -xeu ceph-448a3a16-d543-11ee-95c3-87774f69a715@iscsi.foo.smithi149.gjsccy.service" for details. 2024-02-27T08:00:06.017 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:05 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2386808816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T08:00:06.018 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:05 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2386808816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T08:00:06.041 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-27T08:00:06.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:05 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2386808816' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T08:00:06.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:06 smithi149 ceph-mon[64590]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:06.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:06 smithi149 ceph-mon[67157]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:07.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:06 smithi195 ceph-mon[58095]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:08.003 INFO:teuthology.orchestra.run.smithi149.stdout:true 2024-02-27T08:00:08.623 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-27T08:00:08.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[64590]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:08.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/2263875604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:08.926 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:08.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[67157]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:08.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/2263875604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:08.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:08 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:09.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:08 smithi195 ceph-mon[58095]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:09.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:08 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/2263875604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:09.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:08 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:10.621 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)" 2024-02-27T08:00:10.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:10 smithi149 ceph-mon[67157]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:10.926 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:10 smithi149 ceph-mon[67157]: from='client.? 172.21.15.149:0/730357529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:10.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:10 smithi149 ceph-mon[64590]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:10.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:10 smithi149 ceph-mon[64590]: from='client.? 172.21.15.149:0/730357529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:11.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:10 smithi195 ceph-mon[58095]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:11.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:10 smithi195 ceph-mon[58095]: from='client.? 172.21.15.149:0/730357529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T08:00:11.322 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-27T08:00:12.826 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:12 smithi149 ceph-mon[64590]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:12.826 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:12 smithi149 ceph-mon[67157]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:13.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:12 smithi195 ceph-mon[58095]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:13.263 INFO:teuthology.orchestra.run.smithi149.stdout:osd 8 92s ago - 2024-02-27T08:00:13.853 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T08:00:13.863 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'lock': False, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'verbose': True, 'machine_type': 'smithi', 'os_type': None, 'owner': 'scheduled_adking@teuthology', 'interactive_on_error': False, 'suite_path': None, 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'os_version': None, 'block': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574539', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'mon.a': ['ceph config set mgr mgr/cephadm/use_repo_digest false --force']}}, {'cephadm.shell': {'mon.a': ['radosgw-admin realm create --rgw-realm=r --default', 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default', 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default', 'radosgw-admin period update --rgw-realm=r --commit', 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000', 'ceph orch apply rgw smpl', 'ceph osd pool create foo', 'rbd pool init foo', 'ceph orch apply iscsi foo u p', 'sleep 120', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'mon.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph versions', 'ceph orch upgrade status', 'ceph health detail', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1", "ceph orch ls | grep '^osd '"]}}, {'cephadm.shell': {'mon.a': ['ceph orch upgrade ls', 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0', 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']], [Remote(name='ubuntu@smithi195.front.sepia.ceph.com'), ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCskd1lovrDBBAA0xYDOJpP6SvAF/xPbbtVfw==\n', bootstrap_remote=Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '448a3a16-d543-11ee-95c3-87774f69a715', 'mon election default strategy': 1}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': 'true', 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 448a3a16-d543-11ee-95c3-87774f69a715\n[global]\n\tfsid = 448a3a16-d543-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0]\n', first_mgr='y', first_mon='a', first_mon_role='mon.a', fsid='448a3a16-d543-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCrkd1ls4yDDxAAI5L1lY3CbYkl+aQJJduxzA==\n\tcaps mon = "allow *"\n', mons={'mon.a': '172.21.15.149', 'mon.c': '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'mon.b': '172.21.15.195'}, roleless=False, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'mon.a': ['ceph orch upgrade ls', 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0', 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2']}, 'cluster_name': ''} 2024-02-27T08:00:13.863 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T08:00:13.864 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-27T08:00:15.129 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:08:00:14] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.33.4" 2024-02-27T08:00:15.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[64590]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:15.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[64590]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:15.129 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[64590]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T08:00:15.130 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[67157]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:15.130 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[67157]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:15.130 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:14 smithi149 ceph-mon[67157]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T08:00:15.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:14 smithi195 ceph-mon[58095]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:15.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:14 smithi195 ceph-mon[58095]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:15.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:14 smithi195 ceph-mon[58095]: from='mgr.25336 172.21.15.149:0/142721144' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout: "image": "quay.io/ceph/ceph", 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout: "registry": "quay.io", 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout: "bare_image": "ceph/ceph", 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout: "versions": [] 2024-02-27T08:00:16.435 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-02-27T08:00:17.162 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:16 smithi149 ceph-mon[64590]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:17.162 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:16 smithi149 ceph-mon[64590]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:17.162 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:16 smithi149 ceph-mon[67157]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:17.163 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:16 smithi149 ceph-mon[67157]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:17.189 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-02-27T08:00:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:16 smithi195 ceph-mon[58095]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:17.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:16 smithi195 ceph-mon[58095]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:19.176 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:18 smithi149 ceph-mon[67157]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:19.177 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:18 smithi149 ceph-mon[67157]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:18 smithi149 ceph-mon[64590]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:19.177 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:18 smithi149 ceph-mon[64590]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:18 smithi195 ceph-mon[58095]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:19.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:18 smithi195 ceph-mon[58095]: from='client.54691 -' entity='client.iscsi.foo.smithi149.gjsccy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-27T08:00:19.837 INFO:teuthology.orchestra.run.smithi149.stdout: "16.2.0", 2024-02-27T08:00:20.527 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 448a3a16-d543-11ee-95c3-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-27T08:00:20.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:20 smithi195 ceph-mon[58095]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:20.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:20 smithi195 ceph-mon[58095]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:21.131 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:20 smithi149 ceph-mon[67157]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:21.131 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:20 smithi149 ceph-mon[67157]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:21.131 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:20 smithi149 ceph-mon[64590]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:21.131 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:20 smithi149 ceph-mon[64590]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-27T08:00:23.084 INFO:teuthology.orchestra.run.smithi149.stdout: "v16.2.2", 2024-02-27T08:00:23.084 INFO:teuthology.orchestra.run.smithi149.stdout: "v16.2.2-20210505", 2024-02-27T08:00:23.085 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:22 smithi149 ceph-mon[67157]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:23.085 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:22 smithi149 ceph-mon[64590]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:23.214 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:22 smithi195 ceph-mon[58095]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-27T08:00:23.683 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-27T08:00:23.693 INFO:tasks.cephadm:Teardown begin 2024-02-27T08:00:23.693 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T08:00:23.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T08:00:23.759 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-27T08:00:23.759 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-27T08:00:23.778 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-27T08:00:23.817 INFO:tasks.cephadm:Stopping all daemons... 2024-02-27T08:00:23.817 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-27T08:00:23.817 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a 2024-02-27T08:00:23.927 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:23 smithi149 ceph-mon[67157]: from='client.54802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:23.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:23 smithi149 ceph-mon[64590]: from='client.54802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:23.927 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:23 smithi149 systemd[1]: Stopping Ceph mon.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:24.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:23 smithi195 ceph-mon[58095]: from='client.54802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T08:00:24.426 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[64586]: 2024-02-27T08:00:24.127+0000 7f58ee792700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a[64586]: 2024-02-27T08:00:24.127+0000 7f58ee792700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:24 smithi149 podman[90821]: 2024-02-27 08:00:24.18947887 +0000 UTC m=+0.186105473 container died 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE Bus STOPPING 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE Bus STOPPED 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE Bus STARTING 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: CherryPy Checker: 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: The Application mounted at '' has an empty config. 2024-02-27T08:00:24.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: 2024-02-27T08:00:24.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE Serving on http://:::9283 2024-02-27T08:00:24.428 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: [27/Feb/2024:08:00:24] ENGINE Bus STARTED 2024-02-27T08:00:24.746 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:24 smithi149 podman[90821]: 2024-02-27 08:00:24.57832975 +0000 UTC m=+0.574956361 container cleanup 1865cab475cfb789c2a488c0a5447f50bbd96067a23bb8af5f8cdeaed3f22132 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-27T08:00:24.746 INFO:journalctl@ceph.mon.a.smithi149.stdout:Feb 27 08:00:24 smithi149 bash[90821]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-a 2024-02-27T08:00:24.951 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.a.service' 2024-02-27T08:00:24.999 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y[60289]: ::ffff:172.21.15.195 - - [27/Feb/2024:08:00:24] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.33.4" 2024-02-27T08:00:25.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:25.002 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-27T08:00:25.002 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-27T08:00:25.002 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.c 2024-02-27T08:00:25.300 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 systemd[1]: Stopping Ceph mon.c for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:25.676 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c[67153]: 2024-02-27T08:00:25.299+0000 7f411e91a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:25.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c[67153]: 2024-02-27T08:00:25.299+0000 7f411e91a700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-27T08:00:25.677 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 podman[90956]: 2024-02-27 08:00:25.432954646 +0000 UTC m=+0.253730164 container died b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-27T08:00:26.068 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 podman[90956]: 2024-02-27 08:00:25.799853284 +0000 UTC m=+0.620628815 container cleanup b393c5235d2f168ed826f76e01c1da49e42218e1a34a257844291582d53f0591 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:00:26.068 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 27 08:00:25 smithi149 bash[90956]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-c 2024-02-27T08:00:26.211 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.c.service' 2024-02-27T08:00:26.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:26.249 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-27T08:00:26.249 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-27T08:00:26.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.b 2024-02-27T08:00:26.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:26 smithi195 systemd[1]: Stopping Ceph mon.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:26.823 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b[58091]: 2024-02-27T08:00:26.557+0000 7f8dad1fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:26.823 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b[58091]: 2024-02-27T08:00:26.557+0000 7f8dad1fa700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-27T08:00:27.084 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:26 smithi195 podman[76122]: 2024-02-27 08:00:26.823430383 +0000 UTC m=+0.374395930 container died 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:00:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:27 smithi195 podman[76122]: 2024-02-27 08:00:27.284491747 +0000 UTC m=+0.835457355 container cleanup 2163ea012c4eacbf7eec172ea57a38adf18b5851e114516b658b4134524944c9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-27T08:00:27.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 27 08:00:27 smithi195 bash[76122]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mon-b 2024-02-27T08:00:27.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mon.b.service' 2024-02-27T08:00:27.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:27.805 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-27T08:00:27.806 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-27T08:00:27.806 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y 2024-02-27T08:00:28.154 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:27 smithi149 systemd[1]: Stopping Ceph mgr.y for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:28.427 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:28 smithi149 podman[91090]: 2024-02-27 08:00:28.28526526 +0000 UTC m=+0.287879309 container died c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-27T08:00:28.914 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:28 smithi149 podman[91090]: 2024-02-27 08:00:28.688008562 +0000 UTC m=+0.690622654 container cleanup c790b627593b673b68436342efa8f7d40058b619442486206ab4c994e9678549 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:00:28.914 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Feb 27 08:00:28 smithi149 bash[91090]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-y 2024-02-27T08:00:29.053 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.y.service' 2024-02-27T08:00:29.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:29.093 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-27T08:00:29.093 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-27T08:00:29.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x 2024-02-27T08:00:29.464 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 08:00:29 smithi195 systemd[1]: Stopping Ceph mgr.x for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:29.807 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 08:00:29 smithi195 podman[76243]: 2024-02-27 08:00:29.529909646 +0000 UTC m=+0.239275565 container died 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T08:00:30.190 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 08:00:29 smithi195 podman[76243]: 2024-02-27 08:00:29.91395722 +0000 UTC m=+0.623323180 container cleanup 5408d94e7c8a1d3e7e7a597083743728a2350846e96596c62b060f1273a72d3b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-02-27T08:00:30.190 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 27 08:00:29 smithi195 bash[76243]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-mgr-x 2024-02-27T08:00:30.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@mgr.x.service' 2024-02-27T08:00:30.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:30.358 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-27T08:00:30.359 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-27T08:00:30.359 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0 2024-02-27T08:00:30.668 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:30 smithi149 systemd[1]: Stopping Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:30.926 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T08:00:30.666+0000 7fe3e6221700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:30.926 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T08:00:30.666+0000 7fe3e6221700 -1 osd.0 152 *** Got signal Terminated *** 2024-02-27T08:00:30.926 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:30 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0[70523]: 2024-02-27T08:00:30.666+0000 7fe3e6221700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:00:36.419 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:36 smithi149 podman[91214]: 2024-02-27 08:00:36.154986323 +0000 UTC m=+5.609797836 container died 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-27T08:00:36.676 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:36 smithi149 podman[91214]: 2024-02-27 08:00:36.492464384 +0000 UTC m=+5.947275898 container cleanup 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-27T08:00:36.676 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:36 smithi149 bash[91214]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0 2024-02-27T08:00:37.110 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:36 smithi149 podman[91229]: 2024-02-27 08:00:36.685316441 +0000 UTC m=+0.523759833 container remove 6d66c6d3e8b5eca200f3d0bea97e25bab8c3165d97a82d3ca2aee40065c0d8ac (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:00:37.410 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.01246741 +0000 UTC m=+0.018636733 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:00:37.410 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.127855473 +0000 UTC m=+0.134024784 container create 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-02-27T08:00:37.411 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.322644979 +0000 UTC m=+0.328814303 container init 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=) 2024-02-27T08:00:37.411 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.325980927 +0000 UTC m=+0.332150248 container start 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-27T08:00:37.411 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.410254454 +0000 UTC m=+0.416423779 container attach 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-27T08:00:37.676 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:37 smithi149 podman[91325]: 2024-02-27 08:00:37.628219721 +0000 UTC m=+0.634389022 container died 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T08:00:38.217 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:38 smithi149 podman[91351]: 2024-02-27 08:00:38.201163309 +0000 UTC m=+0.563944046 container remove 3fb049a838237114850370253f2b6c00c16b64961cc52306226ab26b5921363d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T08:00:38.217 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:38 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-27T08:00:38.217 INFO:journalctl@ceph.osd.0.smithi149.stdout:Feb 27 08:00:38 smithi149 systemd[1]: Stopped Ceph osd.0 for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T08:00:38.220 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.0.service' 2024-02-27T08:00:38.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:38.263 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-27T08:00:38.263 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-27T08:00:38.263 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.1 2024-02-27T08:00:38.479 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:38 smithi149 systemd[1]: Stopping Ceph osd.1 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:38.742 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:38 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T08:00:38.639+0000 7f0d09957700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:38.742 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:38 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T08:00:38.639+0000 7f0d09957700 -1 osd.1 152 *** Got signal Terminated *** 2024-02-27T08:00:38.742 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:38 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1[73004]: 2024-02-27T08:00:38.639+0000 7f0d09957700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:00:44.230 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:43 smithi149 podman[91432]: 2024-02-27 08:00:43.979464659 +0000 UTC m=+5.459510591 container died 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:00:44.496 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:44 smithi149 podman[91432]: 2024-02-27 08:00:44.31182142 +0000 UTC m=+5.791867356 container cleanup 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T08:00:44.497 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:44 smithi149 bash[91432]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1 2024-02-27T08:00:44.497 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:00:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:00:44.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:00:44.497 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:00:44 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:00:44.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:00:44.926 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:44 smithi149 podman[91462]: 2024-02-27 08:00:44.496373346 +0000 UTC m=+0.509508066 container remove 989ce187fad23c2d9c973422b59bf1d4661f7c226d4f028d50bde247e5efc9c4 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T08:00:44.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:00:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:00:44.605Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.149:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.149:8765: connect: connection refused" 2024-02-27T08:00:44.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:00:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:00:44.606Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.149:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.149:8765: connect: connection refused" 2024-02-27T08:00:44.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:00:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:00:44.609Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.149:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.149:8765: connect: connection refused" 2024-02-27T08:00:44.964 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:00:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:00:44.609Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.149:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.149:8765: connect: connection refused" 2024-02-27T08:00:44.965 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:00:44 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:00:44.610Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.149:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.149:8765: connect: connection refused" 2024-02-27T08:00:45.241 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:44 smithi149 podman[91553]: 2024-02-27 08:00:44.832314644 +0000 UTC m=+0.018958080 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:00:45.242 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:44 smithi149 podman[91553]: 2024-02-27 08:00:44.941526398 +0000 UTC m=+0.128169825 container create 0aa49190312870669951192358c1e866b92363daa7542da0664b83378734dfb9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:00:45.242 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:45 smithi149 podman[91553]: 2024-02-27 08:00:45.167430612 +0000 UTC m=+0.354074050 container init 0aa49190312870669951192358c1e866b92363daa7542da0664b83378734dfb9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-02-27T08:00:45.242 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:45 smithi149 podman[91553]: 2024-02-27 08:00:45.170869716 +0000 UTC m=+0.357513152 container start 0aa49190312870669951192358c1e866b92363daa7542da0664b83378734dfb9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T08:00:45.242 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:45 smithi149 podman[91553]: 2024-02-27 08:00:45.241617606 +0000 UTC m=+0.428261034 container attach 0aa49190312870669951192358c1e866b92363daa7542da0664b83378734dfb9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux ) 2024-02-27T08:00:45.676 INFO:journalctl@ceph.osd.1.smithi149.stdout:Feb 27 08:00:45 smithi149 podman[91553]: 2024-02-27 08:00:45.468069947 +0000 UTC m=+0.654713382 container died 0aa49190312870669951192358c1e866b92363daa7542da0664b83378734dfb9 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-02-27T08:00:46.023 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.1.service' 2024-02-27T08:00:46.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:46.066 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-27T08:00:46.066 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-27T08:00:46.066 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.2 2024-02-27T08:00:46.373 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:46 smithi149 systemd[1]: Stopping Ceph osd.2 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:46.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T08:00:46.372+0000 7faa3a2cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:46.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T08:00:46.372+0000 7faa3a2cc700 -1 osd.2 152 *** Got signal Terminated *** 2024-02-27T08:00:46.676 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:46 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2[75478]: 2024-02-27T08:00:46.372+0000 7faa3a2cc700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:00:52.000 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:51 smithi149 podman[91657]: 2024-02-27 08:00:51.726698649 +0000 UTC m=+5.437316518 container died 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-02-27T08:00:52.291 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91657]: 2024-02-27 08:00:52.064762584 +0000 UTC m=+5.775380451 container cleanup 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-27T08:00:52.292 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 bash[91657]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2 2024-02-27T08:00:52.677 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91674]: 2024-02-27 08:00:52.291101946 +0000 UTC m=+0.560562427 container remove 5f7f45342b595e473931ca4e669948ea606dafca07dbda170b73526189b4eec7 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-02-27T08:00:52.994 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91766]: 2024-02-27 08:00:52.725182887 +0000 UTC m=+0.114478036 container create 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:00:52.994 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91766]: 2024-02-27 08:00:52.62941195 +0000 UTC m=+0.018707103 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:00:52.994 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91766]: 2024-02-27 08:00:52.927575792 +0000 UTC m=+0.316870941 container init 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-27T08:00:52.994 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91766]: 2024-02-27 08:00:52.930739546 +0000 UTC m=+0.320034699 container start 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-27T08:00:52.994 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:52 smithi149 podman[91766]: 2024-02-27 08:00:52.994199996 +0000 UTC m=+0.383495178 container attach 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T08:00:53.427 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:53 smithi149 podman[91766]: 2024-02-27 08:00:53.216756119 +0000 UTC m=+0.606051262 container died 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:00:53.775 INFO:journalctl@ceph.osd.2.smithi149.stdout:Feb 27 08:00:53 smithi149 podman[91792]: 2024-02-27 08:00:53.772934769 +0000 UTC m=+0.551280651 container remove 586ee059ec9762d139de8e95a80832b050466095358ddc36b00314ddd5b0752b (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=) 2024-02-27T08:00:53.791 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.2.service' 2024-02-27T08:00:53.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:00:53.832 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-27T08:00:53.833 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-27T08:00:53.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.3 2024-02-27T08:00:54.142 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:53 smithi149 systemd[1]: Stopping Ceph osd.3 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:00:54.426 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T08:00:54.141+0000 7f3780a70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:00:54.426 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T08:00:54.141+0000 7f3780a70700 -1 osd.3 152 *** Got signal Terminated *** 2024-02-27T08:00:54.426 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3[77954]: 2024-02-27T08:00:54.141+0000 7f3780a70700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:00:54.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:00:54.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:00:54.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:00:54.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:00:54.427 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:00:54 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:00:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:00:57.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:00:57 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:00:57.146+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:00:58.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:00:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:00:58.142+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:00:58.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:00:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:00:58.562+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:00:58.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:00:58 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:00:58.559+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:00:59.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:00:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:00:59.138+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:00:59.832 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:59 smithi149 podman[91872]: 2024-02-27 08:00:59.522987136 +0000 UTC m=+5.466121873 container died edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-27T08:00:59.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:00:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:00:59.604+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:00:59.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:00:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:00:59.688+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:00:59.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:00:59 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:00:59.556+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:00.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:59 smithi149 podman[91872]: 2024-02-27 08:00:59.850186393 +0000 UTC m=+5.793321127 container cleanup edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-27T08:01:00.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:00:59 smithi149 bash[91872]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3 2024-02-27T08:01:00.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91889]: 2024-02-27 08:01:00.034636875 +0000 UTC m=+0.507967043 container remove edb3d7993300a8bd0b8eb7ece09537367f2b370c6cdeb609e7d654f3b66a962d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-27T08:01:00.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:00.160+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:00.764 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.468732944 +0000 UTC m=+0.112462830 container create 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-27T08:01:00.764 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.374640241 +0000 UTC m=+0.018370128 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:01:00.764 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.663182784 +0000 UTC m=+0.306912673 container init 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T08:01:00.764 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.666301483 +0000 UTC m=+0.310031370 container start 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T08:01:00.764 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.762985533 +0000 UTC m=+0.406715434 container attach 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T08:01:00.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:00.690+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:00.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:00.557+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:00.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:00 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:00.644+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:01.176 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:00 smithi149 podman[91983]: 2024-02-27 08:01:00.955166359 +0000 UTC m=+0.598896248 container died 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T08:01:01.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:01.124+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:01.519 INFO:journalctl@ceph.osd.3.smithi149.stdout:Feb 27 08:01:01 smithi149 podman[92010]: 2024-02-27 08:01:01.516834243 +0000 UTC m=+0.556118486 container remove 3a09a45a71a80e8ce7f1ee41b18ed7ac1f58333a7dcc46cbf010a106ad3fc2b3 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-27T08:01:01.534 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.3.service' 2024-02-27T08:01:01.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:01.575 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-27T08:01:01.575 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-27T08:01:01.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.4 2024-02-27T08:01:01.884 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:01 smithi195 systemd[1]: Stopping Ceph osd.4 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:01.884 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:01.674+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:01.884 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:01.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:01.884 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:01.654+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:02.214 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:01.883+0000 7f347d56e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:01:02.214 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:01.883+0000 7f347d56e700 -1 osd.4 152 *** Got signal Terminated *** 2024-02-27T08:01:02.214 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:01 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:01.883+0000 7f347d56e700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:01:02.214 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:02 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:02.116+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:02.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:02 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:02.575+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:02.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:02 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:02.708+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:02.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:02 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:02.683+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:03.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:03 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:03.115+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:03.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:03 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:03.718+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:03.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:03 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:03.743+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:03.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:03 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:03.553+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:04.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:04.165+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:04.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:01:04.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:04.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:04.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:04 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:04.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:04.540+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:04.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:04.712+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:04.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:04.702+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:04.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:04 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:04.702+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:05.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:05.192+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:05.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:05.502+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:05.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:05.723+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:05.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:05.723+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:05.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:05.682+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:05.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:05 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:05.682+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:06.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:06.172+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:33.494540+0000 front 2024-02-27T08:00:33.494480+0000 (oldest deadline 2024-02-27T08:00:56.394439+0000) 2024-02-27T08:01:06.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4[61373]: 2024-02-27T08:01:06.172+0000 7f3475b5a700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.695524+0000 front 2024-02-27T08:00:41.695619+0000 (oldest deadline 2024-02-27T08:01:05.195386+0000) 2024-02-27T08:01:06.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:06.706+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:06.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:06.706+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:06.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:06.683+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:06.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:06.683+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:06.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:06 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:06.508+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:07.464 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:07 smithi195 podman[76391]: 2024-02-27 08:01:07.206759291 +0000 UTC m=+5.429350979 container died 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-02-27T08:01:07.926 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:07 smithi195 podman[76391]: 2024-02-27 08:01:07.66623371 +0000 UTC m=+5.888825441 container cleanup 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-27T08:01:07.926 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:07 smithi195 bash[76391]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:07.732+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:07.732+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:07.665+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:07.665+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:07.509+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:07.927 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:07 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:07.509+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:08.214 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:07 smithi195 podman[76411]: 2024-02-27 08:01:07.925940427 +0000 UTC m=+0.721332813 container remove 482e1c8ae05fe4186ae176f1ad3e8ddbc8af1cd442bb0153b64ccbbc4516780d (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:08.473+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:08.473+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.186962851 +0000 UTC m=+0.018092090 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.285100677 +0000 UTC m=+0.116229915 container create c9c3a2e2507971858dd1032c7bb0311e1dd347c3ee832b4fd4a7ef5d8e556efa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.529955714 +0000 UTC m=+0.361084967 container init c9c3a2e2507971858dd1032c7bb0311e1dd347c3ee832b4fd4a7ef5d8e556efa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-27T08:01:08.613 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.533213937 +0000 UTC m=+0.364343179 container start c9c3a2e2507971858dd1032c7bb0311e1dd347c3ee832b4fd4a7ef5d8e556efa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-27T08:01:08.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.612834406 +0000 UTC m=+0.443963668 container attach c9c3a2e2507971858dd1032c7bb0311e1dd347c3ee832b4fd4a7ef5d8e556efa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-02-27T08:01:08.964 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 27 08:01:08 smithi195 podman[76504]: 2024-02-27 08:01:08.820787821 +0000 UTC m=+0.651917065 container died c9c3a2e2507971858dd1032c7bb0311e1dd347c3ee832b4fd4a7ef5d8e556efa (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-27T08:01:08.965 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:08.619+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:08.965 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:08.619+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:08.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:08.714+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:08.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:08 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:08.714+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:09.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.4.service' 2024-02-27T08:01:09.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:09.447 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-27T08:01:09.447 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-27T08:01:09.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.5 2024-02-27T08:01:09.650 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 systemd[1]: Stopping Ceph osd.5 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:09.650 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:09.454+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:09.650 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:09.454+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:09.743+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:09.743+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:09.657+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:09.657+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:09.786+0000 7f5de1bfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:09.786+0000 7f5de1bfe700 -1 osd.5 152 *** Got signal Terminated *** 2024-02-27T08:01:09.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:09 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:09.786+0000 7f5de1bfe700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:01:10.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:10.675+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:10.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:10.675+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:10.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:10.704+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:10.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:10.704+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:10.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:10.483+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:10.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:10 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:10.483+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:11.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:11.711+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:11.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:11.711+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:11.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:11.493+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:11.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:11.493+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:11.965 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:11.636+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:11.965 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:11 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:11.636+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:12.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:12.675+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:12.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:12.675+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:12.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:12.647+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:12.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:12.647+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:12.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:12.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:12.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:12 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:13.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:13.687+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:13.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:13.687+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:13.964 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:13.648+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:13.965 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:13.648+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:13.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:13.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:13.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:13 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:14.676 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:01:14.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:14.676 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:14.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:14.676 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:14 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:14.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:14.654+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:14.654+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:14.651+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:35.509976+0000 front 2024-02-27T08:00:35.509858+0000 (oldest deadline 2024-02-27T08:00:59.009960+0000) 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:14.651+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.511880+0000 front 2024-02-27T08:00:43.511916+0000 (oldest deadline 2024-02-27T08:01:04.011765+0000) 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5[63709]: 2024-02-27T08:01:14.651+0000 7f5dda1ea700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:49.312796+0000 front 2024-02-27T08:00:49.312811+0000 (oldest deadline 2024-02-27T08:01:14.012706+0000) 2024-02-27T08:01:14.866 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:14.867 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:14.867 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:15.571 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:15 smithi195 podman[76609]: 2024-02-27 08:01:15.292975392 +0000 UTC m=+5.605062980 container died 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-27T08:01:15.571 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:15.571 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:15.571 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:15.921 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:15 smithi195 podman[76609]: 2024-02-27 08:01:15.696138282 +0000 UTC m=+6.008225868 container cleanup 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:01:15.921 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:15 smithi195 bash[76609]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5 2024-02-27T08:01:15.921 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:15.693+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:15.921 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:15.693+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:15.921 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:15 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:15.693+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:16.214 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:15 smithi195 podman[76637]: 2024-02-27 08:01:15.920599365 +0000 UTC m=+0.624639716 container remove 258882dddff0a552825087c21a0d248fc87752e10ce497d270e5c2241482f1fc (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-27T08:01:16.523 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.164506552 +0000 UTC m=+0.018139955 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:01:16.524 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.271614658 +0000 UTC m=+0.125248060 container create c3a04ce30036a9737ffed67aa064f7ba797b962be81029e3c963adffc40fe87f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-27T08:01:16.524 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.523984088 +0000 UTC m=+0.377617497 container init c3a04ce30036a9737ffed67aa064f7ba797b962be81029e3c963adffc40fe87f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-02-27T08:01:16.809 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:16.715+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:16.715+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:16 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:16.715+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.527099881 +0000 UTC m=+0.380733291 container start c3a04ce30036a9737ffed67aa064f7ba797b962be81029e3c963adffc40fe87f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-27T08:01:16.810 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.623958088 +0000 UTC m=+0.477591492 container attach c3a04ce30036a9737ffed67aa064f7ba797b962be81029e3c963adffc40fe87f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:01:17.102 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 27 08:01:16 smithi195 podman[76729]: 2024-02-27 08:01:16.809719564 +0000 UTC m=+0.663352979 container died c3a04ce30036a9737ffed67aa064f7ba797b962be81029e3c963adffc40fe87f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-27T08:01:17.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.5.service' 2024-02-27T08:01:17.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:17.451 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-27T08:01:17.452 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-27T08:01:17.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6 2024-02-27T08:01:17.714 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 systemd[1]: Stopping Ceph osd.6 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:17.714 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:17.714 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:17.714 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:18.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:01:18.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 osd.6 152 *** Got signal Terminated *** 2024-02-27T08:01:18.214 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:01:18.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:17.757+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:18.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:17.757+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:18.215 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:17 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:17.757+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:18.712+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:18.712+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:18.712+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:18.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:18 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:19.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:19.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:19.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:19.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:19.755+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:19.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:19.755+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:19.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:19 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:19.755+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:20.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:20.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:20.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:20.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:20.720+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:20.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:20.720+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:20.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:20.720+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:20.965 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:20 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:20.720+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:58.235533+0000 front 2024-02-27T08:00:58.235511+0000 (oldest deadline 2024-02-27T08:01:21.035908+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:21.692+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:21.692+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:21.692+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:21.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:21 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:21.692+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:22.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:22.658+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:22.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:22.658+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:22.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:22.658+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:22.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:22.658+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:22.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:22.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:22.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:22.965 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:58.235533+0000 front 2024-02-27T08:00:58.235511+0000 (oldest deadline 2024-02-27T08:01:21.035908+0000) 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: *** Caught signal (Segmentation fault) ** 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:22 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: in thread 7f7de5850700 thread_name:safe_timer 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev) 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e016d4d20] 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x557f2b504f4a] 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x557f2b592d78] 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x557f2b178013] 2024-02-27T08:01:23.357 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x557f2b403f81] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x557f2b0f3d74] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x557f2b0f5dca] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x557f2b0f7d33] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x557f2b058fba] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x557f2b6b320f] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 11: (CommonSafeTimerThread::entry()+0x11) [0x557f2b6b4531] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7e016ca1ca] 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 13: clone() 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:23.000+0000 7f7de5850700 -1 *** Caught signal (Segmentation fault) ** 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: in thread 7f7de5850700 thread_name:safe_timer 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev) 2024-02-27T08:01:23.358 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e016d4d20] 2024-02-27T08:01:23.359 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x557f2b504f4a] 2024-02-27T08:01:23.359 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x557f2b592d78] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x557f2b178013] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x557f2b403f81] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x557f2b0f3d74] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x557f2b0f5dca] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x557f2b0f7d33] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x557f2b058fba] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x557f2b6b320f] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 11: (CommonSafeTimerThread::entry()+0x11) [0x557f2b6b4531] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7e016ca1ca] 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 13: clone() 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -4617> 2024-02-27T08:00:58.562+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.360 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -4447> 2024-02-27T08:00:59.556+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -4374> 2024-02-27T08:01:00.557+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -4158> 2024-02-27T08:01:01.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -4085> 2024-02-27T08:01:02.575+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -3495> 2024-02-27T08:01:03.553+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -3339> 2024-02-27T08:01:04.540+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -3279> 2024-02-27T08:01:05.502+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2730> 2024-02-27T08:01:06.508+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2572> 2024-02-27T08:01:07.509+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2571> 2024-02-27T08:01:07.509+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2519> 2024-02-27T08:01:08.473+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2518> 2024-02-27T08:01:08.473+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2460> 2024-02-27T08:01:09.454+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2459> 2024-02-27T08:01:09.454+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2410> 2024-02-27T08:01:10.483+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2409> 2024-02-27T08:01:10.483+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.361 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2027> 2024-02-27T08:01:11.493+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -2026> 2024-02-27T08:01:11.493+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1931> 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1930> 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1929> 2024-02-27T08:01:12.497+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1820> 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1819> 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1818> 2024-02-27T08:01:13.463+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1376> 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1375> 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1374> 2024-02-27T08:01:14.507+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1302> 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1301> 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1300> 2024-02-27T08:01:15.523+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1208> 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1207> 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.362 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1206> 2024-02-27T08:01:16.537+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1125> 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1124> 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1123> 2024-02-27T08:01:17.584+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1099> 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1098> 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 osd.6 152 *** Got signal Terminated *** 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1096> 2024-02-27T08:01:17.812+0000 7f7dfa27e700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1041> 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1040> 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -1039> 2024-02-27T08:01:18.582+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -856> 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -855> 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -854> 2024-02-27T08:01:19.592+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -802> 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -801> 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -800> 2024-02-27T08:01:20.570+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.363 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -704> 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -703> 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -702> 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -701> 2024-02-27T08:01:21.566+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:58.235533+0000 front 2024-02-27T08:00:58.235511+0000 (oldest deadline 2024-02-27T08:01:21.035908+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -134> 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.933410+0000 front 2024-02-27T08:00:32.933516+0000 (oldest deadline 2024-02-27T08:00:58.233382+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -133> 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:41.134464+0000 front 2024-02-27T08:00:41.134542+0000 (oldest deadline 2024-02-27T08:01:07.034290+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -132> 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:47.034671+0000 front 2024-02-27T08:00:47.034627+0000 (oldest deadline 2024-02-27T08:01:12.334563+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: -131> 2024-02-27T08:01:22.595+0000 7f7df6872700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:58.235533+0000 front 2024-02-27T08:00:58.235511+0000 (oldest deadline 2024-02-27T08:01:21.035908+0000) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 0> 2024-02-27T08:01:23.000+0000 7f7de5850700 -1 *** Caught signal (Segmentation fault) ** 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: in thread 7f7de5850700 thread_name:safe_timer 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev) 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7e016d4d20] 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x557f2b504f4a] 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x557f2b592d78] 2024-02-27T08:01:23.364 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x557f2b178013] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x557f2b403f81] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x557f2b0f3d74] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x557f2b0f5dca] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x557f2b0f7d33] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x557f2b058fba] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x557f2b6b320f] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 11: (CommonSafeTimerThread::entry()+0x11) [0x557f2b6b4531] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7e016ca1ca] 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 13: clone() 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6[66047]: 2024-02-27T08:01:23.365 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 podman[76836]: 2024-02-27 08:01:23.073419007 +0000 UTC m=+5.394006373 container died d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:01:23.644 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 podman[76836]: 2024-02-27 08:01:23.555646982 +0000 UTC m=+5.876234354 container cleanup d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-02-27T08:01:23.644 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 bash[76836]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6 2024-02-27T08:01:23.926 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:23.538Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:23.926 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:23 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:23.538Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 podman[76850]: 2024-02-27 08:01:23.823703269 +0000 UTC m=+0.746477044 container remove d24e22c91a158bb847effebab5ec8f574442f4c1a5a9d23efdbb2cfb684d5bf5 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:23 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:23.643+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:23.643+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:23.643+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:23.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:23 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:23.643+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:24.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.075872205 +0000 UTC m=+0.018116731 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:01:24.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.182720206 +0000 UTC m=+0.124964729 container create d02f14f577dd16e8f4c23adb0b010bdbb48fe7eda3c359a3eb6694a64d1e821f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T08:01:24.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.419033939 +0000 UTC m=+0.361278472 container init d02f14f577dd16e8f4c23adb0b010bdbb48fe7eda3c359a3eb6694a64d1e821f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-27T08:01:24.464 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.422549482 +0000 UTC m=+0.364794008 container start d02f14f577dd16e8f4c23adb0b010bdbb48fe7eda3c359a3eb6694a64d1e821f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T08:01:24.676 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:01:24.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:24.676 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:24.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:24.677 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:24 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:24.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:24.906 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.502021508 +0000 UTC m=+0.444266027 container attach d02f14f577dd16e8f4c23adb0b010bdbb48fe7eda3c359a3eb6694a64d1e821f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-27T08:01:24.907 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 27 08:01:24 smithi195 podman[76943]: 2024-02-27 08:01:24.716833173 +0000 UTC m=+0.659077687 container died d02f14f577dd16e8f4c23adb0b010bdbb48fe7eda3c359a3eb6694a64d1e821f (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-27T08:01:24.907 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:24.613+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:24.907 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:24.613+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:24.907 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:24.613+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:24.907 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:24 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:24.613+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:25.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.6.service' 2024-02-27T08:01:25.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:25.482 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-27T08:01:25.483 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-27T08:01:25.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.7 2024-02-27T08:01:25.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 systemd[1]: Stopping Ceph osd.7 for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:25.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.580+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:25.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.580+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:25.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.580+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:25.714 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.580+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:26.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.791+0000 7fef869b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T08:01:26.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.791+0000 7fef869b6700 -1 osd.7 152 *** Got signal Terminated *** 2024-02-27T08:01:26.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:25 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:25.791+0000 7fef869b6700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T08:01:26.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:26.601+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:26.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:26.601+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:26.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:26.601+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:26.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:26 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:26.601+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:27.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:27 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:27.648+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:27.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:27 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:27.648+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:27.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:27 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:27.648+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:27.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:27 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:27.648+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:28.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:28.652+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:28.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:28.652+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:28.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:28.652+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:28.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:28 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:28.652+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:29.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:29.690+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:29.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:29.690+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:29.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:29.690+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:29.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:29.690+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:29.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:29 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:29.690+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-02-27T08:01:04.628302+0000 front 2024-02-27T08:01:04.628358+0000 (oldest deadline 2024-02-27T08:01:28.728280+0000) 2024-02-27T08:01:31.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:30.730+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6806 osd.0 since back 2024-02-27T08:00:32.424946+0000 front 2024-02-27T08:00:32.424909+0000 (oldest deadline 2024-02-27T08:00:57.724788+0000) 2024-02-27T08:01:31.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:30.730+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6814 osd.1 since back 2024-02-27T08:00:43.025799+0000 front 2024-02-27T08:00:43.025857+0000 (oldest deadline 2024-02-27T08:01:05.325792+0000) 2024-02-27T08:01:31.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:30.730+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6822 osd.2 since back 2024-02-27T08:00:50.527318+0000 front 2024-02-27T08:00:50.527287+0000 (oldest deadline 2024-02-27T08:01:15.227143+0000) 2024-02-27T08:01:31.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:30.730+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.149:6830 osd.3 since back 2024-02-27T08:00:55.227653+0000 front 2024-02-27T08:00:55.227667+0000 (oldest deadline 2024-02-27T08:01:19.927488+0000) 2024-02-27T08:01:31.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:30 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7[68359]: 2024-02-27T08:01:30.730+0000 7fef82faa700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-02-27T08:01:04.628302+0000 front 2024-02-27T08:01:04.628358+0000 (oldest deadline 2024-02-27T08:01:28.728280+0000) 2024-02-27T08:01:31.534 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:31 smithi195 podman[77048]: 2024-02-27 08:01:31.282016952 +0000 UTC m=+5.583488898 container died 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T08:01:31.918 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:31 smithi195 podman[77048]: 2024-02-27 08:01:31.708166041 +0000 UTC m=+6.009637990 container cleanup 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T08:01:31.919 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:31 smithi195 bash[77048]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7 2024-02-27T08:01:32.214 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:31 smithi195 podman[77066]: 2024-02-27 08:01:31.917947998 +0000 UTC m=+0.633101831 container remove 21f14d2033796e6271efd6fc729f68ba7c78a837fe1e0a8e9d5498ef3ed5bdee (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True) 2024-02-27T08:01:32.654 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.227870214 +0000 UTC m=+0.017995198 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T08:01:32.654 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.335434926 +0000 UTC m=+0.125559952 container create 82c408545e824f7c421eec4cac8f856e2c7f3e15a0ba696e884796f5a94ed850 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-02-27T08:01:32.654 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.570867778 +0000 UTC m=+0.360992762 container init 82c408545e824f7c421eec4cac8f856e2c7f3e15a0ba696e884796f5a94ed850 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-27T08:01:32.654 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.573848463 +0000 UTC m=+0.363973453 container start 82c408545e824f7c421eec4cac8f856e2c7f3e15a0ba696e884796f5a94ed850 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-27T08:01:32.654 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.654242029 +0000 UTC m=+0.444367018 container attach 82c408545e824f7c421eec4cac8f856e2c7f3e15a0ba696e884796f5a94ed850 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-27T08:01:32.964 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 27 08:01:32 smithi195 podman[77159]: 2024-02-27 08:01:32.862169217 +0000 UTC m=+0.652294201 container died 82c408545e824f7c421eec4cac8f856e2c7f3e15a0ba696e884796f5a94ed850 (image=quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-02-27T08:01:33.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@osd.7.service' 2024-02-27T08:01:33.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:33.462 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-27T08:01:33.462 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-27T08:01:33.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a 2024-02-27T08:01:33.660 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 systemd[1]: Stopping Ceph prometheus.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:33.926 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:01:33.538Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:33.926 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:33.539Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:33.927 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:33 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:33.539Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.660Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.661Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.661Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a[53957]: ts=2024-02-27T08:01:33.661Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-27T08:01:33.953 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:33 smithi195 podman[77237]: 2024-02-27 08:01:33.670599249 +0000 UTC m=+0.119667280 container died 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:34.214 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:34 smithi195 podman[77237]: 2024-02-27 08:01:34.043522524 +0000 UTC m=+0.492590563 container cleanup 170b78cb18efd2f55d0d9c0486c0adab96a4e712272477da1213922dd8ed402f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:34.214 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:34 smithi195 bash[77237]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus-a 2024-02-27T08:01:34.214 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 27 08:01:34 smithi195 bash[77265]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-prometheus.a" found: no such container 2024-02-27T08:01:34.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-448a3a16-d543-11ee-95c3-87774f69a715@prometheus.a.service' 2024-02-27T08:01:34.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T08:01:34.362 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-27T08:01:34.362 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 448a3a16-d543-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-27T08:01:34.608 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=error ts=2024-02-27T08:01:34.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:34.608 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:34.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-02-27T08:01:34.608 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:34 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=warn ts=2024-02-27T08:01:34.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-27T08:01:36.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:35 smithi149 systemd[1]: Stopping Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:36.178 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:35 smithi149 podman[92377]: 2024-02-27 08:01:35.994091735 +0000 UTC m=+0.100730038 container died 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:36.501 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 podman[92377]: 2024-02-27 08:01:36.325572869 +0000 UTC m=+0.432211187 container cleanup 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:36.501 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 bash[92377]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a 2024-02-27T08:01:36.863 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:36 smithi149 systemd[1]: Stopping Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:01:36.863 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 podman[92391]: 2024-02-27 08:01:36.501669673 +0000 UTC m=+0.509466253 container remove 73faee7fa866ad13475f6b5986e256500ab983ec0dc394dd58f38b789c9f894a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:36.863 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-27T08:01:36.863 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 bash[92403]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.a" found: no such container 2024-02-27T08:01:36.864 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-27T08:01:36.864 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 systemd[1]: Stopped Ceph node-exporter.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T08:01:36.864 INFO:journalctl@ceph.node-exporter.a.smithi149.stdout:Feb 27 08:01:36 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.a.service: Consumed 3.784s CPU time. 2024-02-27T08:01:37.176 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:36 smithi149 ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a[59021]: level=info ts=2024-02-27T08:01:36.862Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-27T08:01:37.176 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:36 smithi149 podman[92457]: 2024-02-27 08:01:36.91417603 +0000 UTC m=+0.156570325 container died 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:37.547 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 podman[92457]: 2024-02-27 08:01:37.316202881 +0000 UTC m=+0.558597175 container cleanup 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:37.547 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 bash[92457]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a 2024-02-27T08:01:37.548 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 bash[92484]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager.a" found: no such container 2024-02-27T08:01:37.827 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 podman[92472]: 2024-02-27 08:01:37.547533098 +0000 UTC m=+0.624544706 container remove 8e272eb9189d424ab72c59f6a07d8620f07ec3c1932a3a0c01092315141f48fd (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-27T08:01:37.827 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 podman[92472]: 2024-02-27 08:01:37.63167758 +0000 UTC m=+0.708689191 volume remove 300b194248c0b6c4933a95a8f062a503b8c06cf89b8a3b3797564d2cdc70ec2d 2024-02-27T08:01:37.827 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-27T08:01:37.827 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 systemd[1]: Stopped Ceph alertmanager.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T08:01:37.827 INFO:journalctl@ceph.alertmanager.a.smithi149.stdout:Feb 27 08:01:37 smithi149 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@alertmanager.a.service: Consumed 1.035s CPU time. 2024-02-27T08:02:11.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 448a3a16-d543-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-27T08:02:13.287 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 systemd[1]: Stopping Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:02:13.544 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 podman[77609]: 2024-02-27 08:02:13.287065351 +0000 UTC m=+0.104757820 container died 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T08:02:13.961 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 podman[77609]: 2024-02-27 08:02:13.735110947 +0000 UTC m=+0.552803461 container cleanup 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T08:02:13.961 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 bash[77609]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b 2024-02-27T08:02:13.961 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 bash[77636]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter.b" found: no such container 2024-02-27T08:02:14.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 systemd[1]: Stopping Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715... 2024-02-27T08:02:14.214 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 podman[77623]: 2024-02-27 08:02:13.961404011 +0000 UTC m=+0.674153509 container remove 54ca3ae0e6d5c77869589522ffb1d6aee444ac4bc7a579c73983f245fc1aff7d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-27T08:02:14.214 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-27T08:02:14.214 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-27T08:02:14.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 systemd[1]: Stopped Ceph node-exporter.b for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T08:02:14.215 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 27 08:02:13 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@node-exporter.b.service: Consumed 3.932s CPU time. 2024-02-27T08:02:14.566 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a[51086]: t=2024-02-27T08:02:14+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-27T08:02:14.566 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 podman[77689]: 2024-02-27 08:02:14.275102318 +0000 UTC m=+0.125698706 container died 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, io.openshift.tags=base rhel8, name=ubi8, com.redhat.component=ubi8-container) 2024-02-27T08:02:14.857 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 podman[77689]: 2024-02-27 08:02:14.647703862 +0000 UTC m=+0.498300247 container cleanup 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., architecture=x86_64, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.openshift.tags=base rhel8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, version=8.5, distribution-scope=public, com.redhat.component=ubi8-container, io.buildah.version=1.24.2) 2024-02-27T08:02:14.857 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 bash[77689]: ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a 2024-02-27T08:02:14.857 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 bash[77719]: Error: no container with name or ID "ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana.a" found: no such container 2024-02-27T08:02:15.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 podman[77703]: 2024-02-27 08:02:14.857198816 +0000 UTC m=+0.584030217 container remove 9914d359e30216a82798270cc76e54a51a6fd8d28606f2848bc1d05a0a725dc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-448a3a16-d543-11ee-95c3-87774f69a715-grafana-a, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8.5, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, architecture=x86_64, io.buildah.version=1.24.2, vcs-type=git, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-27T08:02:15.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-27T08:02:15.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 systemd[1]: Stopped Ceph grafana.a for 448a3a16-d543-11ee-95c3-87774f69a715. 2024-02-27T08:02:15.214 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 27 08:02:14 smithi195 systemd[1]: ceph-448a3a16-d543-11ee-95c3-87774f69a715@grafana.a.service: Consumed 4.120s CPU time. 2024-02-27T08:02:37.910 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T08:02:37.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T08:02:37.976 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-27T08:02:37.978 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi149/crash 2024-02-27T08:02:37.978 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash -- . 2024-02-27T08:02:38.035 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-27T08:02:38.035 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2024-02-27T08:02:38.037 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi195/crash 2024-02-27T08:02:38.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash -- . 2024-02-27T08:02:38.077 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/448a3a16-d543-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-27T08:02:38.077 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-02-27T08:02:38.078 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-27T08:02:38.078 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | head -n 1 2024-02-27T08:02:38.110 INFO:teuthology.orchestra.run.smithi149.stdout:2024-02-27T07:51:40.059228+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T08:02:38.111 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-27T08:02:38.111 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[SEC\]' /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | head -n 1 2024-02-27T08:02:38.181 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[ERR\]' /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | head -n 1 2024-02-27T08:02:38.251 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[WRN\]' /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | head -n 1 2024-02-27T08:02:38.320 INFO:teuthology.orchestra.run.smithi149.stdout:2024-02-27T07:51:40.059228+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-27T08:02:38.320 INFO:tasks.cephadm:Compressing logs... 2024-02-27T08:02:38.320 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-27T08:02:38.363 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:: No such file or directory 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.a.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi149.bmexvh.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.a.log: /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi149.bmexvh.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.1.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr: 76.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi149.bmexvh.log.gz 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.3.log 2024-02-27T08:02:38.391 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log: /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi149.akhivq.log 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.y.log 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi149.akhivq.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/tcmu-runner.log 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr: 76.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi149.akhivq.log.gz/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.y.log: 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.0.log 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr: 84.3% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/tcmu-runner.log.gz 2024-02-27T08:02:38.392 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log 2024-02-27T08:02:38.394 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log: 90.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-27T08:02:38.394 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.2.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:: No such file or directory 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.6.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi195.lvkuxs.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi195.lvkuxs.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.4.log 2024-02-27T08:02:38.395 INFO:teuthology.orchestra.run.smithi195.stderr: 77.2% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.smpl.smithi195.lvkuxs.log.gz 2024-02-27T08:02:38.401 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.7.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.5.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.4.log: /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.x.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log: 83.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log: 90.5% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.b.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log: 87.3% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi195.xuyrge.log 2024-02-27T08:02:38.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.b.log: 88.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-27T08:02:38.403 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log: 94.2% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-27T08:02:38.403 INFO:teuthology.orchestra.run.smithi149.stderr: 92.9% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-27T08:02:38.403 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-27T08:02:38.404 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.c.log 2024-02-27T08:02:38.404 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-27T08:02:38.406 INFO:teuthology.orchestra.run.smithi149.stderr: 91.3% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-27T08:02:38.410 INFO:teuthology.orchestra.run.smithi195.stderr: 91.4% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-27T08:02:38.410 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi195.xuyrge.log: 76.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-client.rgw.foo.smithi195.xuyrge.log.gz 2024-02-27T08:02:38.452 INFO:teuthology.orchestra.run.smithi195.stderr: 88.9% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.x.log.gz 2024-02-27T08:02:38.716 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.c.log.gz 2024-02-27T08:02:38.736 INFO:teuthology.orchestra.run.smithi149.stderr: 88.6% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mgr.y.log.gz 2024-02-27T08:02:38.810 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.b.log.gz 2024-02-27T08:02:39.287 INFO:teuthology.orchestra.run.smithi149.stderr: 91.0% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-mon.a.log.gz 2024-02-27T08:02:39.345 INFO:teuthology.orchestra.run.smithi149.stderr: 93.3% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.2.log.gz 2024-02-27T08:02:39.400 INFO:teuthology.orchestra.run.smithi195.stderr: 93.5% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.6.log.gz 2024-02-27T08:02:39.683 INFO:teuthology.orchestra.run.smithi195.stderr: 93.6% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.5.log.gz 2024-02-27T08:02:39.778 INFO:teuthology.orchestra.run.smithi149.stderr: 93.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.1.log.gz 2024-02-27T08:02:39.778 INFO:teuthology.orchestra.run.smithi195.stderr: 94.0% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.7.log.gz 2024-02-27T08:02:39.856 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.4.log.gz 2024-02-27T08:02:39.857 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-02-27T08:02:39.858 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m1.480s 2024-02-27T08:02:39.858 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m5.453s 2024-02-27T08:02:39.858 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.225s 2024-02-27T08:02:39.911 INFO:teuthology.orchestra.run.smithi149.stderr: 93.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.0.log.gz 2024-02-27T08:02:40.160 INFO:teuthology.orchestra.run.smithi149.stderr: 93.7% -- replaced with /var/log/ceph/448a3a16-d543-11ee-95c3-87774f69a715/ceph-osd.3.log.gz 2024-02-27T08:02:40.162 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-27T08:02:40.162 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m1.787s 2024-02-27T08:02:40.162 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m6.978s 2024-02-27T08:02:40.162 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.254s 2024-02-27T08:02:40.163 INFO:tasks.cephadm:Archiving logs... 2024-02-27T08:02:40.163 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi149/log 2024-02-27T08:02:40.164 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-27T08:02:41.145 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi195/log 2024-02-27T08:02:41.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-27T08:02:41.937 INFO:tasks.cephadm:Removing cluster... 2024-02-27T08:02:41.938 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 448a3a16-d543-11ee-95c3-87774f69a715 --force 2024-02-27T08:02:42.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 448a3a16-d543-11ee-95c3-87774f69a715 --force 2024-02-27T08:02:42.499 INFO:tasks.cephadm:Removing cephadm ... 2024-02-27T08:02:42.499 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-27T08:02:42.515 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-27T08:02:42.529 INFO:tasks.cephadm:Teardown complete 2024-02-27T08:02:42.530 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-27T08:02:42.540 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-27T08:02:42.540 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T08:02:42.557 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T08:02:42.571 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +666us[ +691us] +/- 131ms 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -349us[ -323us] +/- 51ms 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +130us[ +130us] +/- 69ms 2024-02-27T08:02:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T08:02:42.584 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +679us[ +679us] +/- 131ms 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -377us[ -351us] +/- 50ms 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +159us[ +159us] +/- 69ms 2024-02-27T08:02:42.588 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T08:02:42.588 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-27T08:02:42.597 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-27T08:02:42.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-27T08:02:42.639 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-27T08:02:42.720 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-27T08:02:42.797 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T08:02:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T08:02:42.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T08:02:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T08:02:42.858 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-27T08:02:42.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-27T08:02:42.888 INFO:teuthology.task.internal:Duration was 1699.136859 seconds 2024-02-27T08:02:42.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-27T08:02:42.922 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-27T08:02:42.922 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-27T08:02:42.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-27T08:02:42.966 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T08:02:42.967 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T08:02:43.464 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-27T08:02:43.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-02-27T08:02:43.464 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-27T08:02:43.524 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-02-27T08:02:43.525 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-27T08:02:43.549 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-27T08:02:43.549 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-27T08:02:43.567 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-27T08:02:43.622 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-27T08:02:43.622 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-27T08:02:43.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-27T08:02:44.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-27T08:02:44.071 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-27T08:02:44.071 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-27T08:02:44.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-27T08:02:44.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-27T08:02:44.134 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-27T08:02:44.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-27T08:02:44.163 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-02-27T08:02:44.190 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-02-27T08:02:44.219 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-27T08:02:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T08:02:44.238 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-27T08:02:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T08:02:44.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-27T08:02:44.283 INFO:teuthology.task.internal:Transferring archived files... 2024-02-27T08:02:44.284 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi149 2024-02-27T08:02:44.284 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-27T08:02:44.329 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/remote/smithi195 2024-02-27T08:02:44.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-27T08:02:44.370 INFO:teuthology.task.internal:Removing archive directory... 2024-02-27T08:02:44.370 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-27T08:02:44.372 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-27T08:02:44.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-27T08:02:44.429 INFO:teuthology.task.internal:Not uploading archives. 2024-02-27T08:02:44.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-27T08:02:44.447 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-27T08:02:44.448 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-27T08:02:44.450 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-27T08:02:44.462 INFO:teuthology.orchestra.run.smithi149.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 27 08:02 /home/ubuntu/cephtest 2024-02-27T08:02:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: 266032 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 27 08:02 /home/ubuntu/cephtest 2024-02-27T08:02:44.473 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-27T08:02:44.482 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-27T08:02:44.547 INFO:teuthology.nuke:Checking targets against current locks 2024-02-27T08:02:44.577 DEBUG:teuthology.nuke:shortname: smithi149 2024-02-27T08:02:44.577 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T08:02:44.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 07:24:53.398304', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE3T1OerhF6G0+GX4dL1Lzc1U7ROJ+d8anXw2claJ3lhmhnfLRBA3nmN6Ku5qksz1yyz3blSvSdqUECDJjFkwmo='} 2024-02-27T08:02:44.604 DEBUG:teuthology.nuke:shortname: smithi195 2024-02-27T08:02:44.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T08:02:44.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 07:24:53.400082', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo5AL9AzrlKjvQJd7zFMbX8hzHKq26gXbsZFtLzUcKlHtcKx2M7smjmHaRgRGnkKUDSvwcS+3MRg/fYH2ami3c='} 2024-02-27T08:02:44.661 INFO:teuthology.orchestra.console.smithi149:Power off 2024-02-27T08:02:44.662 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-27T08:02:44.688 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-02-27T08:02:44.688 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:44.701 INFO:teuthology.orchestra.console.smithi195:Power off 2024-02-27T08:02:44.702 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-27T08:02:44.710 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-02-27T08:02:44.723 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-02-27T08:02:44.724 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:44.744 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-27T08:02:48.712 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:48.745 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:48.822 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-02-27T08:02:48.854 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-27T08:02:52.826 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:52.855 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:52.935 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-02-27T08:02:52.963 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-02-27T08:02:53.063 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-02-27T08:02:56.940 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T08:02:57.050 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-02-27T08:02:57.151 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-02-27T08:02:57.254 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1699.136859178543 failure_reason: '"2024-02-27T07:51:40.059228+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_adking@teuthology success: false 2024-02-27T08:02:57.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-27T08:02:57.335 INFO:teuthology.run:FAIL